User:Daniel Kinzler (WMDE)/Evolving the MediaWiki Architecture (DevSummit 2018)

From mediawiki.org

Resources:[edit]

Keep in mind:[edit]

  • Architecture impacts 3rd parties, supporting 3rd parties impacts architecture
  • Top-down (vision) vs. bottom-up (issues)
  • Scale and sustain vs. evolve and explore
  • Session outcomes should inform annual plan as well as movement strategy
  • How do we develop a tech strategy of there is no product strategy yet?

Desired Outcomes:[edit]

  • Urgent Focus for Scaling and Sustaining
    • Identify risks to and needs for sustaining and scaling our current capabilities (aka services)
  • Strategic Direction for Improvement
    • Key Questions blocking Development Decisions (decision tree)
    • Strategic Direction and Technological Visions (with justification and dependency on key questions)
  • Strategic Planning Process
    • Define further process and responsibility for decision making for each focus area / key question
    • Define convergence points that ensure product and tech strategy match

Session Structure[edit]

  • Define session scope, clarify desired outcomes, present agenda
  • Discuss Focus Areas
    • Discuss and Adjust. Note that we are not trying to come to a final agreement, we are just prioritizing and assigning responsibilities!
    • For each proposition
      • Decides whether there is (mostly) agreement or disagreement an the proposition(s).
      • Decide whether there is more need for discussion on the topic, and how urgent or important that is.
      • Identify any open questions that need answering from others, and from who (product, ops, etc)
      • Decides who will drive the further discussion/decision process. A four months deadline is suggested.
  • Discuss additional strategy questions. For each question:
    • Decide whether it is considered important, blocking tech strategy.
    • Discuss who should answer it.
    • Decide who will follow up on it.
  • Wrap up

Session Topics[edit]

These topics were compiled by Daniel Kinzler based on the feedback on T183313 and related discussions on IRC, the ATWG, and various documents linked above.

API first (isolate presentation layer) [10 minutes][edit]

Proposition:  for any application logic, provide an internal service interface (PHP) and an external binding (HTTP API). A “presentation layer” is implemented on top of either of those.

Context:

  • Provides “knowledge as a service”
  • The “presentation layer” covers all PHP generation: skins, special pages, action=xyz output, even (final) parser output (the parser may generate annotated HTML, but DOM processing applies).
  • Presentation layer can be PHP (and use the internal interface) or JS (using the HTTP API). For migration, one can emulate the other (JS can call API to get HTML, PHP can call JS to get HTML)
  • Special pages would be implemented as presentations of an underlying API. As a migration step, and perhaps for providing the necessary input forms, annotated HTML could be used, similar to the one emitted by the parser.
  • Page composition
    • Inside index.php entry point for a vanilla install.
    • For WMF, “at the edge” with ESI. Possibly on the client with ServiceWorkers or similar?
    • Get components from MediaWiki for now
    • Allows bit-by-bit migration to “JS-Only” UI if desired
  • REST vs action API mainly lies in the way parameters are formatted, and how routing works. Supporting “nice” REST URLs on top of existing API code should not be hard.
  • RESTbase: There should be a single entry point for interacting with MediaWiki (possibly supporting multiple API flavors). Having two API stacks supporting different functionality seems undesirable. However, some API requests could be routed to RESTbase (via MediaWiki, or at the Varnish layer). That way, the caller no longer needs to know whether a given functionality is implemented in PHP or in JS, and we can change the implementation at will.
  • Note that the above applies to read operations via the API. API design for write operations are governed by different concerns.
    • How does API design impact Multi-DC capability (routing)?
  • “API first” is required by the Isomorphic JS strategy, but it’s also useful if we decide against that.

Decide:

  • Is there agreement for the proposition (yes, no, contingent)
  • Is this a highly relevant topic to be addressed in the tech strategy?
  • Who will take care of turning the proposition into a strategy goal? Until when? If nobody feels able, who will find somebody who is able?
  • What are the relevant open questions to be answered by others that block decision making on the proposition? Who should answer them?

Mobile First (Isomorphic JS) [10 min][edit]

Proposition: All presentation layer code should be written in a way that allows it to run on the client or the server (Isomorphic JavaScript).

Context:

  • Requires node.js on the server, for all installs!
  • This means no support for vanilla LAMP, that means that we no longer run on shared hosting platforms.
    • We would cut off the “long tail” of mini-wikis that don’t have a voice or lobby.
    • ...or we develop separate plain PHP UI based on the same which we don’t use, and hand over to the community for maintenance.
    • ...or we convince at least a few big shared hosting providers to offer node.js
    • ...or we make a hybrid approach possible: require either node.js on the server (for supporting low-tech clients), or require “modern” JS on the client, but no node.js on the server.
  • Alternative: Progressive Web Apps (PWA). Essentially means that “good” JS is required on the client for anything other than reading.

Decide:

  • Is there agreement for the proposition (yes, no, contingent)
  • Is this a highly relevant topic to be addressed in the tech strategy?
  • Who will take care of turning the proposition into a strategy goal? Until when? If nobody feels able, who will find somebody who is able?
  • What are the relevant open questions to be answered by others that block decision making on the proposition? Who should answer them?

Unified Parser Stack [10min][edit]

Proposition: We want to replace Parser.php with (something like) Parsoid.

Context:

  • Having two parsers is bad. Parsoid is the better parser architecture (though it’s not fully compatible with the old parser, and not as performant).
  • We can: a) call parsoid.js from PHP b) port Parsoid to PHP (not that hard, PEG.js can generate PHP and C as well as JS).
  • Allows wikitext content to be made available as a parse tree (resp annotated HTML) via the API for further processing. Final HTML is generated by the presentation layer (final HTML may differ by user language, use options, target device, etc)
  • This unification would also make it easier to work on self-contained templates and similar “wikitext 2.0” features.
  • All parser output would use the same caching mechanism via the ParserCache interface. This may or may not be backed by Cassandra (which is currently used for caching parsoid output).
  • We could still keep the old parser as option. We could use content_model to identify the parser to be used for a given content object.

Decide:

  • Is there agreement for the proposition (yes, no, contingent)
  • Is this a highly relevant topic to be addressed in the tech strategy?
  • Who will take care of turning the proposition into a strategy goal? Until when? If nobody feels able, who will find somebody who is able?
  • What are the relevant open questions to be answered by others that block decision making on the proposition? Who should answer them?

MediaWiki core refactoring [10 min][edit]

Proposition: Refactor mediawiki core code to improve modularization using Dependency Injection (DI), removing access to global state (including configuration) entirely.

Context:

  • Refactoring should be informed by principles of Domain Driven Design where useful/applicable.
  • Modularization improves re-use, maintainability, testability
  • Reduces technical debt
  • Enables cross-wiki logic: with full DI, storage- and presentation services for multiple wikis can co-exist in a single request, e.g. allowing content from multiple wikis to be generated and presented together, removing boundaries between projects and communities (-> “cross-project collaboration” session).
  • selective SOA: Services can be implemented internally as well as externally.
    • This can be used to improve security (e.g. have a separate service for password checks).
    • What would we want SOA for? What would we not want it for? Why?
  • Improve extension interface stability:
    • Confine Gadgets to using stable APIs (like Firefox add-ons, etc). Note: this is a pretty big topic, and deserves more attention elsewhere - perhaps in the “technical community” session.
    • Narrow signatures for hooks (passing $this considered harmful)
  • Evolution Strategy:
    • Rewrite or Refactor?
      • Rewrite has advantages, but often doesn’t lead to replacement, because it fails to address migration.
    • All improvements need a migration path!
    • Deprecation vs. Emulation

Decide:

  • Is there agreement for the proposition (yes, no, contingent)
  • Is this a highly relevant topic to be addressed in the tech strategy?
  • Who will take care of turning the proposition into a strategy goal? Until when? If nobody feels able, who will find somebody who is able?
  • What are the relevant open questions to be answered by others that block decision making on the proposition? Who should answer them?

MediaWiki Product [5 min][edit]

Proposition: The MediaWiki platform is a product. Its users are developers. This should be represented in organizational structures and processes, and considered in resourcing.

Context:

  • It’s bad if “everybody does their own plumbing”
  • Ensures consistency in the code base
  • Enables prioritisation of platform improvement vs feature development

Decide:

  • Is there agreement for the proposition (yes, no, contingent)
  • Is this a highly relevant topic to be addressed in the tech strategy?
  • Who will take care of turning the proposition into a strategy goal? Until when? If nobody feels able, who will find somebody who is able?
  • What are the relevant open questions to be answered by others that block decision making on the proposition? Who should answer them?

Scale large database tables [5 min][edit]

Proposition: Several aspects of our database schema don’t scale well for very large wikis. Addressing this issue should be part of our short- and mid-term tech strategy.

Context:

  • revision (split off rev_comment and rev_user_text); could be sharded if we use a different table to list user contributions. Same for the new content and slot tables.
  • recentchanges
  • change_tag
  • log
  • …

Decide:

  • Is there agreement for the proposition (yes, no, contingent)
  • Is this a highly relevant topic to be addressed in the tech strategy?
  • Who will take care of turning the proposition into a strategy goal? Until when? If nobody feels able, who will find somebody who is able?
  • What are the relevant open questions to be answered by others that block decision making on the proposition? Who should answer them?

Scale usage tracking and change propagation [5 min][edit]

Proposition: Usage tracking and change propagation don’t scale well for very large wikis and cross-wiki operations. Addressing this issue should be part of our short- and mid-term tech strategy.

Context:

  • Usage tracking in SQL is hard to scale (see wbc_entity_usage)
  • Re-generating content is hard to scale: it’s slow, de-duplication is hard, granularity is not fine enough.
  • Increasing need for cross-wiki functionality (reading-lists/watchlists, discussions/notifications, wikidata re-use, global Lua modules)
  • A single edit can cause millions of pages to be invalidated.

Decide:

  • Is there agreement for the proposition (yes, no, contingent)
  • Is this a highly relevant topic to be addressed in the tech strategy?
  • Who will take care of turning the proposition into a strategy goal? Until when? If nobody feels able, who will find somebody who is able?
  • What are the relevant open questions to be answered by others that block decision making on the proposition? Who should answer them?

Consolidate Storage Services [5 min][edit]

Proposition: Consolidate blob storage services to reduce operational overhead and improve knowledge sharing.

Context:

  • We have various use cases for persistent, semi-persistent and volatile/caching storage for blobs.
  • Currently, we have a “zoo” of technologies used for storage (mysql, swift, cassandra, memcached, etc)
  • The requirements for the various uses cases should specified and consolidated, allowing us to reduce the zoo to what we really need.

Decide:

  • Is there agreement for the proposition (yes, no, contingent)
  • Is this a highly relevant topic to be addressed in the tech strategy?
  • Who will take care of turning the proposition into a strategy goal? Until when? If nobody feels able, who will find somebody who is able?
  • What are the relevant open questions to be answered by others that block decision making on the proposition? Who should answer them?

Product Strategy Questions [10min][edit]

These questions should be discussed in addition to the questions that came up during the discussion of the focus areas above.

  • Machine Translation -> (perhaps in the “Future of Languages” session)
    • Do we need it? What for? Content? Discussions?
    • Can we build it ourselves? If not, who do we want to depend on?
    • Can we host it ourselves?  If not, who do we want to depend on?
  • Content evolution:
    • Are we assuming the main way to contribute will remain to be text? By typing?
    • Are we expecting a big increase in multimedia content?
  • 3rd party wiki owners:
    • Who owns the decision on supported platforms? Who decides in how far we want to support shared hosting?
    • Who owns the 3rd party installation process?
      • Easy install for 3rd parties (-> “third party support” session)
      • Easy setup of developers (-> “growing the community” session)
      • Vagrant? Docker? Kubernetes?...
      • How do we manage libraries (PHP and JS), extensions, and services?
  • User maintained code:
    • Who owns the experience of extension developers?
    • Who owns the experience of bot authors?
    • Who owns the experience of gadget authors?
    • Who owns the experience of Lua authors?

Decide for each:

  • Decide whether it is considered important, blocking tech strategy.
  • Discuss who should answer it.
  • Decide who will follow up on it.
Retrieved from "https://www.mediawiki.org/w/index.php?title=User:Daniel_Kinzler_(WMDE)/Evolving_the_MediaWiki_Architecture_(DevSummit_2018)&oldid=2698155"