Wikimedia Developer Summit/2018/Evolving the MediaWiki Architecture

From mediawiki.org

Dev Summit '18 https://etherpad.wikimedia.org/p/devsummit18

Evolving the MediaWiki Architecture

DevSummit event[edit]

  • Day & Time: Monday, 2:10 pm – 3:10 pm AND 4:00 pm – 4:30 pm
  • Room: Tamalpis
  • Facilitator: Erika
  • Lead: Daniel/Corey
  • Notetaker(s): Matt, Roan.

Keep in mind:[edit]

  • Architecture impacts 3rd parties, supporting 3rd parties impacts architecture

Kunal: Overall requirements for MediaWiki product affect specific parts like parser.

  • Top-down (vision) vs. bottom-up (issues)
  • Scale and sustain vs. evolve and explore
  • Session outcomes should inform annual plan as well as movement strategy
  • How do we develop a tech strategy of there is no product strategy yet?

Desired Outcomes:[edit]

  • Strategic Direction for Improvement
    • Key Questions blocking Development Decisions (decision tree)
    • Strategic Direction and Technological Visions (with justification and dependency on key questions)
  • Strategic Planning Process
    • Define further process and responsibility for decision making for each focus area / key question

Session Structure[edit]

  • Define session scope, clarify desired outcomes, present agenda
  • Discuss Focus Areas
    • Discuss and Adjust. Note that we are not trying to come to a final agreement, we are just prioritizing and assigning responsibilities!
    • For each proposition
      • Why or why not (5 minutes): two people expressing, plus direct responses
      • Identify any open questions that need answering from others, and from who (product, ops, etc)
      • Decide whether there is (mostly) agreement or disagreement an the proposition(s) (yes, no, contingent)
  • Discuss additional strategy questions. For each question:
    • Decide whether it is considered important, blocking tech strategy.
      • Identify dependencies
      • Relative importance / urgency of the issues.
      • Why does something block tech strategy
  • Wrap up

Session Topics and Notes[edit]

These topics were compiled by Daniel Kinzler based on the feedback on T183313 and related discussions on IRC, the ATWG, and various documents linked above.


API first (isolate presentation layer) [10 minutes][edit]

Proposition: for any application logic, provide an internal service interface (PHP) and an external binding (HTTP API). A “presentation layer” is implemented on top of either of those.

Context:

  • Provides “knowledge as a service”
  • The “presentation layer” covers all PHP generation: skins, special pages, action=xyz output, even (final) parser output (the parser may generate annotated HTML, but DOM processing applies).
  • Presentation layer can be PHP (and use the internal interface) or Node.js (using the HTTP API). For migration, one can emulate the other (Node.js can call API to get HTML, PHP can call Node.js to get HTML)
  • Special pages would be implemented as presentations of an underlying API. As a migration step, and perhaps for providing the necessary input forms, annotated HTML could be used, similar to the one emitted by the parser.
  • Page composition
    • Inside index.php entry point for a vanilla install.
    • For WMF, “at the edge” with ESI. Possibly on the client with ServiceWorkers or similar?
    • Get components from MediaWiki for now
    • Allows bit-by-bit migration to “JS-Only” UI if desired
      • Important for Wikidata – it currently uses a JS based UI anyway, but it is old and it would be great if it would match the rest of the WM universe
  • REST vs action API mainly lies in the way parameters are formatted, and how routing works. Supporting “nice” REST URLs on top of existing API code should not be hard.
  • RESTbase: There should be a single entry point for interacting with MediaWiki (possibly supporting multiple API flavors). Having two API stacks supporting different functionality seems undesirable. However, some API requests could be routed to RESTbase (via MediaWiki, or at the Varnish layer). That way, the caller no longer needs to know whether a given functionality is implemented in PHP or in JS, and we can change the implementation at will.
  • Note that the above applies to read operations via the API. API design for write operations are governed by different concerns.
    • How does API design impact Multi-DC capability (routing)?
  • “API first” is required by the Isomorphic JS strategy, but it’s also useful if we decide against that.

Decide:

  • Why or why not (5 minutes): two people expressing, plus direct responses
  • Identify any open questions that need answering from others, and from who (product, ops, etc)
  • Decide whether there is (mostly) agreement or disagreement an the proposition(s) (yes, no, contingent)

[API first] Notes

  • Corey: Do we have someone to say we we'd want to do that?
  • Adam: So, there are downsides to doing this sort of thing of course. I would generally prefer we focus on this particular problem. It's the thing that probably works best for multi-device support. There are thorny pieces to this, which Tim can speak to. There's been a lot of discussion in past years about letting APIs power things like progressive web apps. Or progressive enhancement. We might need to select first interfaces to migrate to this architecture.
  • Tim: Wanted to channel my internal Brad Jorsch here, we've talked about this a fair bit.
  • The consensus we came to is that you should have an internal service/class interface to a particular logic or functionality, and on top of that an HTTP API module and also perhaps PHP presentation;
  • but if there's PHP presentation that doesn't have to be built on top of the HTTP API. It should be built on some sort of internal sensibly structured class system.
  • Corey: Those are the two broad reasons [which?] why we'd want to do that. Is there something that we've missed?
  • Matt: This is an issue that's been disputed. In some cases you want a [PHP] class structure backing it up either way, but it's been disputed whether for simpler APIs it's OK to just have the API module and call that from PHP, or whether you should have a class structure for everything.
  • Dan A: One thing we're missing is there's other stuff we might want to integrate that doesn't come out of PHP necessarily, like Analytics, streams of data, composings of scores of quality, or whatever you can think of. Makes sense to expose as a service and consume in UI, not rewrite? in PHP.
  • Dan A: We're talking about 2030, it's a long time from now, the vision is hard but spitballing, surface bias, surface people's opinions on scores that are given to revisions by machine learning, to act on editor level. So a few revisions, want to know if they should revert or not. More integrated into interface in fluid way, scores, etc. It's complicated, interface needs to more complex than PHP hitting MySQL. Change how we think about data, more fundamentally than just the API part.
  • Timo: When I see the text here, two different approaches come to mind. Not sure if both are in scope. One is exposing the business logic and our content in a way that can be consumed by services, apps, and by ourselves as a page composition service. Mainly about exposing the contentwithout entanglement with the skin, can the user edit this, etc. A very different thing is everything in MW PHP being in an API module and being consumed as such internally. That's a very different topic, seems out of scope to me but want to clarify.
  • Corey: How does exposing MW functionality as an API, how is that in conflict with making public APIs?
  • Timo: Not necessarily in conflict in terms of implementation, but for the discussion here they're orthogonal. Are we talking about refactoring things in terms of API modules, or about enabling a page composition service? (One is a problem we can solve, the other is an imho overly generic solution with multiple separate problems it could solve)
  • Subbu: I wonder if this is an issue of -- what is the MW product? Is analytics and revision scoring something that needs to be exposed to all MW installations, or just WMF? That influences whether it needs to be ported back to PHP (post-comment thought: assuming that is the direction we are taking) or can stay in a separate service like it is right now.
  • Brion: Just responding to Timo's idea that this is two separate issues, they are but they go to same goals. Especially when it comes to not just page content, but also interactions with software available to people. Particularly special pages and edit actions. It's very awkward to recreate those in a separate UI. They're very tied to web interface, being able to make them exposed to additional UI is essential to getting not just reader experience, but full citizen experience in other UIs.
  • Giuseppe: Was about to say basically what Brion said. Question for Tim: if you don't separate business from presentation very sharply, is it easy to expose everything to third-party skin, another service, same as core component. That's the problem you usually have when you don't have this separation.
  • Tim: That's basically correct. Yes, there are risks that business logic is not exposed if you don't have strict separation. Question is, is that a price we're willing to pay in exchange for the other considerations. [Tim typing:] You can add API modules as needed, so you can mitigate the problem of missing APIs that way. Certainly you can add every API needed for a PCS if that's what we want to do.
  • Matt: Disagree w/ the idea of a separate design based on whether it's WMF-specific. Everything should be available to everyone and possible to reasonably use. Even if it's WMF-specific now, it might not be in the future, so we shouldn't design based on that.
  • Corey: We're at time. Do want to get something out of the discussion: there might be a question we need to asnwer. Does anyone have an unknown or a question about this that would help us decide and move forward?
  • Adam: You're asking us to formulate a question that will be answered later, right?
  • James: Yes.
  • Derk-Jan: Do we want to create a possibility where everyone can create an editor, publish it somewhere, and use and publish wiki content? Do we want that for ourselves, and do we want that for others? If we answer that question we'll understand whether we want APIs.
  • Matt: Definitely we do want that. Our mission is to share the world's knowledge, we can't do that alone. Users of these other platforms also need to be able to contribute back, an essential part of the mission. In order to enable that, edit APIs need to be rock solid and include everything, so everyone can upstream.
  • Erika: Need to move on to the next topic.
  • Adam: Question that we ought to pose to ourselves: as an engineering culture, are we willing to enforce an API-first stance in our code reviews? That's a question we ought to think about.
  • Markus: I still don't have a good answer to the question what the benefit is of switching to this model.

What are the next steps? If we want this, who's repsonsibility should it be to formulate this as a strategy item?

  • individual features can refactor themselves at will, but if this is a Thing We Need then we need to specify a standard api-first stance (techcom's responsibility, maybe?) and then teams need to work on it. probably there'd be some coordination with the first implementations of alternate UIs (audiences/mobile).
  • [brion] I think the driver should be WMF Audiences (Adam maybe? :D), as they really want the separate UI frontends. Thus things will be tested first in their interfaces. -> in service/product terminology they're the client


Unified Parser Stack [15min][edit]

Proposition: We want to replace Parser.php with (something like) Parsoid.

Context:

  • Having two parsers is bad. Parsoid is the better parser architecture (though it’s not fully compatible with the old parser, and not as performant).
  • Two choices:
    1. Call parsoid.js from PHP
    2. Port Parsoid to PHP (not that hard, PEG.js can generate PHP and C, as well as JS). [ Subbu: it is not "that hard" :-) PEG.js is a small part of the codebase. Async code paths are part of the codebase and so it cannot be a simple port unless more work is done on the node.js version. We can do it, but we should do it carefully and not assume it is simple. ]
  • Allows wikitext content to be made available as a parse tree (resp annotated HTML) via the API for further processing. Final HTML is generated by the presentation layer (final HTML may differ by user language, use options, target device, etc)
  • This unification would also make it easier to work on self-contained templates and similar “wikitext 2.0” features.
  • All parser output would use the same caching mechanism via the ParserCache interface. This may or may not be backed by Cassandra (which is currently used for caching parsoid output).
  • We could still keep the old parser as option. We could use content_model to identify the parser to be used for a given content object.

Decide:

  • Why or why not (5 minutes): two people expressing, plus direct responses
  • Identify any open questions that need answering from others, and from who (product, ops, etc)
  • Decide whether there is (mostly) agreement or disagreement an the proposition(s) (yes, no, contingent)

[Unified Parser] Notes:

  • Corey: This is pretty simple to articulate. Do we want to unify parser stack? Should we have committed to decomission one of the parsers? Do we just want one (parser)?
  • Ariel: Does Parsoid have a published spec that allows people to implement that same parser in some other language?
  • James: mw.org/wiki/spec [1]
  • James: Back to nominations for speaking in favor and/or against.
  • Subbu: I think we definitely want a single parser. The question I think is, what form the single parser takes. It's very clear Parsoid is the implementation (not considering whether it's node.js or PHP or something else), Parsoid as a parsing technology is the future for wikitext. Will it be ported back to PHP, stay a separate service? +1
  • Kunal: Agree with everything he said, additional considerations are mainly how extensions tie into Parsoid. A few native extensions, otherwise calls into MW, it's not a full parser, more like half a parser. Preprocessor also uses core parser. Want all of those to be in same language, not talk over HTTP. Constantly going back and forth, not efficient.
  • Subbu: I think there are two approaches there, one is port more things into nodejs, the other is port Parsoid back into PHP. For extensions, I think independently of whether Parsoid is in PHP or nodejs, extension APIs will have to change, because of how different Parsoid is.
  • Giuseppe: Simply from an infrastructure perspective, the separation doesn't make sense. My other point is that I don't think that there's any good reason to separate the parser out from core. I need to see a use case where that separation makes sense because it can be used for another functoin that has nothing to do with editing/parsing MW pages. Separate services should be justified with additional functioality or a security reason, otherwise it's a maintenance nightmare.
  • Adam: Clarification: you're talking about specifically stuff running on the same host, or not? Or is it about using the same language or the same request?
  • Giuseppe: I don't see a reason to do any form of interprocess communication for parsing content, unless there's a security issue (edge cases can cause user input to cause vulnerabilities), or we want to reuse the service for something else. If there's no such resaon, I strongly argue against separation.
  • Adam: So it might be written in a modular way, but still in the same request/response lifecycle?
  • Giuseppe: Yes, unless there's a strong reason not to.
  • Timo: It seems not everyone agrees on a single parser, but I propose a slightly less ambitious goal: you're able to parse wikitext in an environment where there is only one parser, even if other parsers exist. Right now, you can parse wikitext with Parsoid, but once you hit certain kinds of content, it needs the PHP parser. You can't consistently use a single syntax if the parsers are different. If we want to maintain both indefinitely, we need to get to a point where any embedded wikitext within needs to be handled by the same parser, not the other one.
    • Timo: Proposed goal: Able to parse a blob of wikitext in a single parser (with regards to edge cases and bugs that differ between parsers), so if Parsoid.js does need to use MW.php for handling an extension tag, it would need to be limited to returning parsed html, not invoking another parser. If it does need to run another parser, it should pass the next bit of wikitext back to the same parser as where it started, never a case of running PHP parser within Node.js parser (like we do now, e.g. wikitext from Lua is handled by PHP parser, even if the Lua call originated from Parsoid.js).
  • Max: I don't think you'll be able to make MW work with an external parser. In addition to page text, there are lots of smaller things like messages (you can have to parse dozens of them for a page). You could batch this, but MW's design doesn't take this into account. You'll end up with lots of ad-hoc parsing and be killed by latency.
  • Joel: Anyone who has not spoken?
  • Corey: Want to get a question out of this.
  • Matt: Is there any reason to keep Parsoid in nodejs? (Besides the obvious one that porting to PHP is work.)
    • Tim: Performance.
    • James: Node is faster than PHP right now.
  • Giuseppe: Let's stop talking about PHP vs nodejs and start talking about functionality.
    • Matt [not spoken]: It's a relevant question because having all parsing functionality in one language is simpler, I'm not inherently "anti-node" or "pro-node".
    • (Note: PEG.js can generate PHP and C. could re-use parsoid grammar. | is it written in PEG now? | Yes, Parsoid is written in PEG)
  • Matanya: Would like to generalize the question to be "should we move to one parser".
  • Subbu: Anyone disagree? Does anyone think we should have two parsers forever? [ No hands raised ]
  • Sam: If the PHP parser is required for messages, and Parsoid can't be made fast enough to unify them, or the other way around, it seems a bit intractable.
  • James: But that's an engineering problem. (crowd laughs)
  • Subbu: Performance is a main reason, we've already talked about prototyping a port. If you want to port the preprocessor to node, that's also possible. But it's an architecture question: do we want preprocessing code in node or PHP? Messages need this. The unresolved question is hte archtecture boundary. Do we want the paresr, all of it, preprocessing, extensions, tidying, in nodejs or in PHP. That's the core question that hasn't been resolved. Not a question about language, but about architecture. As a team, we are committed to prototyping it, we still don't have direction as to what is it that we want to get to in the end.
  • Ariel: I wanna hear what we thought we were going to get by moving Parsoid out. Don't care about languages, just that it's a separate service. We did that for a reason, what was it? Did we get that out of it, is it valuable to preserve? If we didn't, how do we get it? +100 +42 +999
  • [Subbu responding after the fact: This was before my time, but here is what I understand. It was NOT possible to write this in PHP in 2012 when this started. There was no HTML5 parser, performance was a concern because of all the additional work that needed to happen. There was PEG.js available. So, the separation wasn't necessarily on a whim but because writing it in PHP wasn't feasible at that time. I don't know if I am speculating here, but there was some passing idea / thought of possibly running Parsoid in the browser and node.js was what enabled it. So, if we had to do it all in PHP, Parsoid, and VE launch might have taken much longer. The original Parsoid design had (and still has remnants) an integrated preprocessor and wasn't meant to be a separate call into M/W api. The call to M/W api happened in the rush to have things ready for the Dec 2012 VE launch and since that worked, it stuck and we didn't pursue the preprocessor integration beyond that. Also see https://www.mediawiki.org/wiki/Parsing#E:_Evaluate_feasibility_of_porting_Parsoid_to_PHP Note that, RemexHTML, the HTML5 parser in PHP is based off the node.js HTML5 parser, domino, which is something we switched to around 2014 from another HTML5 parser. ]

MediaWiki core refactoring [15 min][edit]

Proposition: Refactor mediawiki core code to improve modularization using Dependency Injection (DI), removing access to global state (including configuration) entirely.

Context:

  • Refactoring should be informed by principles of Domain Driven Design where useful/applicable.
  • Modularization improves re-use, maintainability, testability
  • Reduces technical debt
  • Enables cross-wiki logic: with full DI, storage- and presentation services for multiple wikis can co-exist in a single request, e.g. allowing content from multiple wikis to be generated and presented together, removing boundaries between projects and communities (-> “cross-project collaboration” session).
  • selective SOA: Services can be implemented internally as well as externally.
    • This can be used to improve security (e.g. have a separate service for password checks).
    • What would we want SOA for? What would we not want it for? Why?
  • Evolution Strategy:
    • Rewrite or Refactor?
      • Rewrite has advantages, but often doesn’t lead to replacement, because it fails to address migration.
    • All improvements need a migration path!
    • Deprecation vs. Emulation
  • Improve extension interface stability: [JanD: Did I put it there? It does not seem to fit well]
    • Confine Gadgets to using stable APIs (like Firefox add-ons, etc). Note: this is a pretty big topic, and deserves more attention elsewhere - perhaps in the “technical community” session. [JanD.: For how the API looks like etc. it makes sense there, but for technology it might make more sense to put it close to UI] Yes, you are right. Unrelated to refactoring the PHP code. This is an artifact of me frantically merging topics last night :)
    • Narrow signatures for hooks (passing $this considered harmful)

Decide:

  • Why or why not (5 minutes): two people expressing, plus direct responses
  • Identify any open questions that need answering from others, and from who (product, ops, etc)
  • Decide whether there is (mostly) agreement or disagreement an the proposition(s) (yes, no, contingent)

[MW Core refactoring] Notes:

  • Kunal: I can just say what Daniel would say if he was here: we need to use DI and remove global state so our code is more testable and we regress less often. Can be implemented in MW itself, don't need HTTP requests as abstraction layer. Should really improve structure of MW. Would a third-party developer be able to understand flow without spending months. If so, you've achieved your goal.
  • Dan A: This is not "the" question to answer, but perhaps a useful question from an outsider. Can we get to a place where the code is so clear that we remove all the magic so I can write a module calling out to other data sources and get that out to the UI, so we don't need to have a million data sources serving different data?.
  • Brion: I have a clarifying question regarding what we mean by refactoring. We usually mean improvements that don't affect how the code works, they clean up how the code looks, how you interact. But as a developer, we're created extensions, front-end stuff, back-end stuff plugging in. What is the impact on people not on MW team of this kind of refactoring.
  • Tim: I was trying to think, "Who is going to speak in favor of global state?", we're all on board with DI now, since we've introduced a service container with all the global state. All of the things that used to use global variables are in the container and you get them from there. Maybe he'd like the container to only be used at the entry points. Service instantiation only at the entry points. I don't know if that's feasible, or extensible. We have a hook about saving an edit. At what point do you pass into that hook. THat hook happens to call AbuseFilter, creating variables for user-supplied script to execute on. Pulling in a lot of global state to inject into AF. Daniel doesn't want kitchen sink architecture.
    • Matt [not spoken]: Intermediate goal would be to only do it at all entry points, including hook listeners.
    • RK [ns]: The container is global/static, it gives you instances of objects ("services") that are themselves constructed using DI.
  • Daniel: DI is not just for making the code nicer. It allows "selective SOA", and it allows for fully integrated cross-wiki functionality
  • Daniel: CSCott wants to be able to load and render pages from to wikis side by side. With DI, that's trivial.
  • Leszek: Devil's advocate for global state. Do we believe that any kind of refactoring can take us away from the horrible monolith that MW is now and enable cool things that Daniel talked about. Isn't it more likely that we'll get what we want by rethinking MW entirely.
  • Gergo: I think everyone can agree that half of MW uses DI and the other half uses global state, that's the current situation and it doesn't seem to be improving. Had an RFC about DI two years ago, don't seem to have made huge progress. If we're never going to get there, should we just give up and go back? And maybe it's a more general question than DI, how do we ensure that these decisions get carried out in a reasonable amount of time? In the meantime we're making it harder to understand the code.
  • James: Can people propose blocking questions to ask now and answer later?
  • Sam: Other part of this proposition was about config, which I think there's more agreement on.
  • Matanya: What's blocking us and what's the cost?
  • Timo: We've used the services container for a while, it seems agreed that we're going in that direction, but how far are we going to take it? One proposal would be to go the extreme way and require the injection of everything. But that means that if you need a service deep down, you have to pass it down a lot of layers. We could just agree that we're going to do that.
  • Kunal: Main blocker has been resourcing, getting devs to focus on tech debt. Also when fixing something we should take the time to fix the code around it. But that only works if you touch the code around somrething for another reason, so we need resourcing on just cleaning up parts of the code.
  • AdamS: Do we want this as an initial step? There are things that were merged in MW core in the last year that didn't use MWServices. CommentStore doesn't! I can't mock a CommentStore. Do we want to enforce services for new code?
  • DK [ns]: @Leszek: refactoring the horrible monolith step by step is feasible. A full rewrite is not. +1 (from hoo) +1
    • Leszek: Got it. I was more thinking in the direction of replacing some particular pieces of old MW with something new, piece by piece, branching by abstraction kind of thing. This could be seen as a complete rewrite, I agree. It would be incremental though, not a full-on rewrite all things approach.
  • DK [ns]: @Timo: You never need to pass services through lots of layers. Only services need services, and they are injected via th econstructor. And all services are created by the wiring file. only one layer. always. (Caveat: this is top level services. You may need to inject a factory, get a secondary service from that, and use it. But that's not so horrible)
  • [^ <brion> for the "extension needs access to other stuff" case, a good wiring system may be relevant for plugging in things to get from factories. but I'm less certain what to do for, say, an extension that needs another extension. But probably this is a totally clear problem if I see it from the right DI-first perspective. :D]
  • DK [ns] @brion: An extension needing a service defined by another extension isn't a problem. They all live in the same container. Static hook handler functions are a problem, because you can't inject anything into them, they have to rely on static access to the container.
  • [ <brion> excellent. away with static handlers! service wiring forevah ]



MediaWiki Product [15 min][edit]

Proposition: The MediaWiki platform is a product. Its users are developers. This should be represented in organizational structures and processes, and considered in resourcing.

Context:

  • It’s bad if “everybody does their own plumbing”
  • Ensures consistency in the code base
  • Enables prioritisation of platform improvement vs feature development

Decide:

  • Is there agreement for the proposition (yes, no, contingent)
  • Is this a highly relevant topic to be addressed in the tech strategy?
  • Who will take care of turning the proposition into a strategy goal? Until when? If nobody feels able, who will find somebody who is able?
  • What are the relevant open questions to be answered by others that block decision making on the proposition? Who should answer them?
  • Matanya: The question is whether we want to do it. I think we want to do it, but not in this structure.
  • ?: It would be cool to define things.
  • Tim Starling: "Users as developers" implies it's an API.
    • DK [not here]: In the wider sense, yes - developers interact with MediaWiki core, by using the services it defines, by using with extension points, etc.
  • Kunal: Based on what Matanya was saying, "MW platform" means software stack (core, extensions, maybe including VE, RESTBase). I don't agree that users are developers. Users are everyone, but not limited to developers.
  • David Barratt: Since technology is a byproduct, not our mission, it's a byproduct, not a product, according to our mission.
  • Mark Hershberger: I would just push back that just as there is a Wiktionary project, there is a MW.org project that is part of the core mission.
  • Ariel: I'm looking at the first developer who worked on that platform. They worked on it because it was a product, not just because of the mission. If we can't sell it and get more developers, it'll die, that's what happens when something is a second-class citizen.
  • Gergo: If we are still serious about this approach...
  • James F: We aren't splitting, because there are no rooms to go to.
  • Kunal: Can we focus on 2? topics, starting with what is MW?
  • Giuseppe: So, I think there is a fundamental disagreement about what we'll talk about today. MW infrastructure, WMF infra, or the details of how the PHP problem that's called MW works. The problem that I see is the position papers linked, and the ticket suggests we should focus on ? infrastructure as a whole.
  • Corey: This is very nebulous. For others, we have slides, e.g. about APIs and very concrete topics. Do we feel more comfortable going to those topics.
    • DK: The strategic question is whether we want to support MediaWiki as a product in it's own right, officially. And if so, what that means: Is it an application? Or a framework? How owns th experience of extension developers?
  • Erika: Can I just hold for one second? There are 4 other topics. Should we break into 4 groups, talk about together for rest of time, then report back at end?
  • [Voting on session procedure]
  • Corey: You all want to stay together?
  • [Stay here wins]
  • Sj: Can you encourage people to share thoughts in Etherpad? Unlike speaking time, etherpad space is ~non-rival. So if noone gets to you when you want to speak, don't worry :)
  • James F, etc.: Consider everyone welcome to do so.
  • Corey: Let's start this topic. API first.
  • + Is the MediaWiki platform identical w/ MediaWiki as an installable tool, for individuals and groups?
  • + [thinks of MW Foundation debates] -- are the priorities of the extended universe of MW users/hackers/extenders a top priority here? Or are they expected to adapt to whatever ends up being most useful to Wikimedia projects?
  • Ex: I've heard many people say of MW sites that are pegged at an old version, "they're not really MediaWiki users; that site's not really running MediaWiki. They made some mods that relied on old pieces of architecture, so they haven't updated in over 3 years!" ----> this line only makes sense for a certain narrow concept of "MW user".


Mobile First (Isomorphic JS) [15 min][edit]

Proposition: All presentation layer code should be written in a way that allows it to run on the client or the server (Isomorphic JavaScript). [basically: Frontend in JS + Server side rendering in Node.js]

Context:

  • Requires node.js on the server, for all installs [? see hybrid approach]
  • This means no support for vanilla LAMP, that means that we no longer run on shared hosting platforms (which typically don't have node.js)
    • We would cut off the “long tail” of mini-wikis that don’t have a voice or lobby.
    • ...or we develop separate plain PHP UI based on the same which we don’t use, and hand over to the community for maintenance.
    • ...or we convince at least a few big shared hosting providers to offer node.js
    • ...or we make a hybrid approach possible: require either node.js on the server (for supporting low-tech clients), or require “modern” JS on the client, but no node.js on the server.
  • Alternative: Progressive Web Apps (PWA). Essentially means that “good” JS is required on the client for anything other than reading.

Decide:

  • Is there agreement for the proposition (yes, no, contingent)
  • Is this a highly relevant topic to be addressed in the tech strategy?
  • Who will take care of turning the proposition into a strategy goal? Until when? If nobody feels able, who will find somebody who is able?
  • What are the relevant open questions to be answered by others that block decision making on the proposition? Who should answer them?
  • Why does that mean LAMP needed? As far as I understood, as long as JS in enabled client side we could also render on the client. It is slower and you need JS, but it would enable small wikis without node.js
    • See "hybrid approach". | Ahhh. That makes the "No LAMP" seem much less dramatic! | If it's possible. I'm not sur. But I really like the idea :) | I never developed for a large platform, but that is my standard setup: Only client side rendering. Seems also to be a sign of a good architecture if it works both ways (would it be isomorphic if it does not work?) |

Notes

  • Brion: Does this apply to everything in the current regular frontend, or specific to an alternate front-end that will be created in the future by someone?
  • Corey: Scope is for all functionality.
  • Kunal: Are we discussing this in theory, or a specific implementation like Marvin?
  • James: In theory.
  • Corey: Marvin was a prototyping attempt to do this.
  • Giuseppe: You could use C++ instead of JS with WebAssembly, not JS-specific.
  • Moriel: Focusing question, I think isomorphic is awesome, but not for everything, and not all of MW. When you say "everything", I disagree. There are products where the best implemnetation might be isomorphic, and ones where it's not.
  • Corey: Maybe you can answer why would you support it in the context where you would support it?
  • CScott: (View1) As another reason why we might want isomorphic JS, for community purposes, we have a huge number of people out of in the world who know JS. As an Xtreme example: JS everywhere! Why not Scribunto modules in JS! Extensions in JS! Gadgets in JS! (We already have those. If tyof you think it's good for front-end and back-end, why not go all in and say "you only have to learn one language, which everyone teaches anyway, and then you can do things all over our stack".
  • JanD: I could provide the wikidata viewpoint as an exta viewpoint for this.
  • Moriel: (View2) Huge topic, but I'll try to give a couple things. When I think about isomorphic, it depends on which product. Certain ones, we see an inhernt diff between no-JS experience and JS experience. Then there's no point in isomorphic. The product can split between no-JS and JS experiences. Ther eare several products that require a deep connectoin and as much equality as possible between the no-JS and JS versions. Tthne isomorphic saves a lot of time and effort, and deduplication of code. Example off the top of my head: structured discussions. Something to work on a talk page in a continuous way, has to be something that works for everyone, so need a good no-JS. Has to have as many features as possible in no-JS while having a good JS expreience. So you could program something for no-JS in PHP and another thing in JS, but eventually we are [...]ing the no-JS experience, whereas if you go isomorphic, every time you include the JS you can also improve the no-JS at very little cost.
  • JanD: (View3) My point from the Wikidata perspecitve. Wikidata currently already has a frontend mainly in JS, here what's interesting is what we currently can't do with that framework. Built on an old paradigm, uses the DOM directly a lot, we have several problems. One of them, jQuery UI is deprecated, want to get away from that. Another is that the development paradigm is very particular to that custom built structure, would be good to move with the rest of the world to React or Vue to have more exchange. Re isomorphic, wikidata is suited to mini-contirbutions but doesn't work well on mobile. Super hacky, looks bad. Would like to have a good mobile experience.
  • Hard to move forward with integration of JS, up-to-date toolchain, will it work with all our MW magic. There's a lot of FUD which needs structure. Gadgets are also a challenge, since they would need API to the DOM (since modern MVVMs flush DOM) . Those are the resaons Wikidata is currently looking at this.
  • Roan: Something interested to hear from those who spoke -- we've talked about isomorphic JS , sometimes useful, sometimes not... but when you don't need it (no divergence in no-JS exp) how useful/good does it seem to use react/vue/etc?
  • Dan A: Could you give some specific examples?
  • Roan: extreme example is VE -- there is no no-JS experience. other things, they may be so small it's not worth sharing code. in those circumstances, what do proposers think about whether react/vue should be used or not?
  • Moriel: In engineering we tend to fall in love with a language and want to fix everything iwth that language... every language, every stack has its own pluses and minuses... There are things react is amazing at .... [?] could help us reduce load, etc... but some things may not be as good. VE is good example -- even if we rewrote it from scratch may not choose to do it in react for various reasons. Have the consideration that we do have code and odn't want to rewrite everything from scratch... often we have 2/3 options to choose, which is the best one? React can be a solution, in areas where it's good -- example notification popup it would have been perfect for. Elimination of no-JS is an issue... other products, it may not be ideal. Generally: pick things we want to use, not what we're stuck with.
  • Dan A: What Moriel said is eminently sensible, but I do want to push back a little bit on our assumptions about our products. What if, 10 years down the line, I'm editing graphs, tabular data, and other rich content with the aid of Visual Editor extensions, and the doorbell rings. I run down, come back, my editing session that took me a long time to set up is lost. It would be nice to hit 'save', and when I come back it renders back for me in the exact same way on the server. So, I think we should plan not for how our products work now, but how they might work in the near future. I'm sure there are very good examples of products where server-rendering of complex UI will not be needed for the foreseable future. But we should carefully question whether or not that's really true, going a bit deeper than we are.
  • CScott: Direct response: crazy five-point plan on what to do. There should be a place to store temporary data... (saving) -> save in your fork until it's done etc.... Response to roan: not proposing we run everything in a single VM service or anything, but making sure we have a consistent API for things. API should be same that parsoid uses server-side; same interfaces you'd use to write an extension, write a module in JS, etc... parsoid has access to some site info api etc....
  • Adam B: Gonna ask a controversial and strange question. I came to WMF to work on WP0, for low-end phones by and large. Aside from first read of a narticle, do we really need to have a non-JS experience.
  • Kunal: Yes. Aside from low-end devices that don't have resources for JS, the biggest pro-no-JS argument these days is privacy and that JS on a bunch of web sites will do ridiculously crazy things. The best way to protect your privacy is to disable JS. Wikimedia movement has a strong investment in privacy, one of the ways we express that is to allow people to at least read and contribute basic content without JS.
  • Adam B: So your value judgment is that this overrides [other things]? [JanD: That depends on the other things…. It is important to some parts, particularly of the editor community BUT to judge about that we would need some statistics…]
  • Kunal: Yes
  • Nick: (not spoken) notes here: https://www.mediawiki.org/wiki/No-JavaScript_notes ← Thanks!
  • Gergo: There's this theory that we cna just switch to JS and we'll have one language. In practice, we'll write English Wikipedia in isomorphic JS and it'll work beautifully, then we'll run into Chinese Wikipedia using language converter, Finnish Wikipedia using FlaggedRevs, Wikibooks using OCR, Wiktionary uses something different, then we go to desktop and have special pages. If you touch those, a significant and vocal part of the community will yell at you. People are used to Vector and Monobook or whatever their favorite skin is. We end up with two different skinning systems, one in JS and PHP, and now we have a similar problem to the two parsers problem.
  • Marius: How much of that shared code would be useful beyond HTML templating, and how much server-side overhead is there in calling out to nodejs, which would outweigh the savings of not having to reimplement things or using a shared template engine for both JS and PHP.
  • Corey: seems to imply very large technical investment that we don't know the scope of, how would we work through the problems of different projects, different languages, special pages.
  • Brion: We cannot answer any of these questions without first answering the quetsion of an API-driven frontend. If you don't have that you cannot do this. Second question, if we do that, do we do this? Third question is do we have the resources to get it done?
  • Corey: Brion makes a good point, a lot of these depend on each other. Can't decide on isomorphic JS without deciding on API-driven front-end.

Adam: API-driven frontend also serves other use cases. CScott: An API-driven system gives us flexibility to try out many different things, one of which is isomorphic JS.

  • DanielK (please voice): I think one main question is [towards what?][towards "do we want a JS-only presentation layer"] whether we want to implement Special Page presentation twice, once in PHP and once in JS. Or whether we want to keep relying on post-processing HTML. That doesn't *have* to be terrible if the initial HTML is good, and has sufficient annotations. [Not an expert on this, but post processing HTML sounds tricky?][That's what mobile frontend does right now. It's horrible.][Whould "implementing special pages" mean: Having HTML with some template-y things between?][that, or something more similar to parsoid output]
    • RK: I think it's impractical to implement *everything* in JS, but I think a single-page app in JS would be quite useful. A lot of things like special pages would still be PHP-backed, but the outer shell of the web site as well as complex features such as Wikidata editing, Flow discussions and notifications rendering could use React/Vue.
    • DK: the application logic of a special page would be written in php, and exposed via an http api. The question is: how and where is the presentation layer implemented? For showing results, but also for input forms for parameters. A hybrid would be: have a PHP presentation layer that emits annotated HTML that can easily be sliced and diced by JS.


  • Open Points currently for Wikidata:
    • Which framework to choose: Vue/React
    • Toolchain: Can we go with Webpack? Can we go with npm? Does Resource Loader work for us
    • How can we run Server Side Rendering services?


  • Roan: I agree with this, but why is it called "mobile first"? It doesn't seem related to mobile at all. +1 +1
    • DK: because the moditivation for going for a JS based front end was mainly to improve the experience on mobile devices. I was looking for a catchy name. Change it ;)
      • Yeah, asking this above there, too. I also think it should be framed in what it means in terms of technology (JS UI+Server side rendering?)

Summary [15 min][edit]

  • Identify dependencies
  • Relative importance / urgency of the issues:
  • What do we need to work on first and why?
  • Why does something block tech strategy?


Resources


Formatted version on-wiki: https://www.mediawiki.org/wiki/User:Daniel_Kinzler_(WMDE)/Evolving_the_MediaWiki_Architecture_(DevSummit_2018)

Topics moved out[edit]

Desired Outcomes:[edit]

  • Urgent Focus for Scaling and Sustaining
    • Identify risks to and needs for sustaining and scaling our current capabilities (aka services)
    • Define convergence points that ensure product and tech strategy match


Product Strategy Questions [10min][edit]

These questions should be discussed in addition to the questions that came up during the discussion of the focus areas above.

  • Machine Translation -> (perhaps in the “Future of Languages” session)
    • Do we need it? What for? Content? Discussions?
    • Can we build it ourselves? If not, who do we want to depend on?
    • Can we host it ourselves? If not, who do we want to depend on?
  • Content evolution:
    • Are we assuming the main way to contribute will remain to be text? By typing?
    • Are we expecting a big increase in multimedia content?
  • 3rd party wiki owners:
    • Who owns the decision on supported platforms? Who decides in how far we want to support shared hosting?
    • Who owns the 3rd party installation process?
      • Easy install for 3rd parties (-> “third party support” session)
      • Easy setup of developers (-> “growing the community” session)
      • Vagrant? Docker? Kubernetes?...
      • How do we manage libraries (PHP and JS), extensions, and services?
  • User maintained code:
    • Who owns the experience of extension developers?
    • Who owns the experience of bot authors?
    • Who owns the experience of gadget authors?
    • Who owns the experience of Lua authors?

Decide for each:

  • Decide whether it is considered important, blocking tech strategy.
  • Discuss who should answer it.
  • Decide who will follow up on it


Scale large database tables [5 min][edit]

Proposition: Several aspects of our database schema don’t scale well for very large wikis. Addressing this issue should be part of our short- and mid-term tech strategy.

Context:

  • revision (split off rev_comment and rev_user_text); could be sharded if we use a different table to list user contributions. Same for the new content and slot tables.
  • recentchanges
  • change_tag
  • log
  • …

Decide:

  • Why or why not (5 minutes): two people expressing, plus direct responses
  • Identify any open questions that need answering from others, and from who (product, ops, etc)
  • Decide whether there is (mostly) agreement or disagreement an the proposition(s) (yes, no, contingent)

Scale usage tracking and change propagation [5 min][edit]

Proposition: Usage tracking and change propagation don’t scale well for very large wikis and cross-wiki operations. Addressing this issue should be part of our short- and mid-term tech strategy.

Context:

  • Usage tracking in SQL is hard to scale (see wbc_entity_usage)
  • Re-generating content is hard to scale: it’s slow, de-duplication is hard, granularity is not fine enough.
  • Increasing need for cross-wiki functionality (reading-lists/watchlists, discussions/notifications, wikidata re-use, global Lua modules)
  • A single edit can cause millions of pages to be invalidated.

Decide:

  • Why or why not (5 minutes): two people expressing, plus direct responses
  • Identify any open questions that need answering from others, and from who (product, ops, etc)
  • Decide whether there is (mostly) agreement or disagreement an the proposition(s) (yes, no, contingent)

Consolidate Storage Services [5 min][edit]

Proposition: Consolidate blob storage services to reduce operational overhead and improve knowledge sharing.

Context:

  • We have various use cases for persistent, semi-persistent and volatile/caching storage for blobs.
  • Currently, we have a “zoo” of technologies used for storage (mysql, swift, cassandra, memcached, etc)
  • The requirements for the various uses cases should specified and consolidated, allowing us to reduce the zoo to what we really need.

Decide:

  • Why or why not (5 minutes): two people expressing, plus direct responses
  • Identify any open questions that need answering from others, and from who (product, ops, etc)
  • Decide whether there is (mostly) agreement or disagreement an the proposition(s) (yes, no, contingent)
Retrieved from "https://www.mediawiki.org/w/index.php?title=Wikimedia_Developer_Summit/2018/Evolving_the_MediaWiki_Architecture&oldid=2699463"