User:Daniel Kinzler (WMDE)/MCR-PO

While refactoring the MediaWiki storage layer for MCR, questions have arisen regarding the handling of multiple slots in ParserOutput.

This document is intended for discussing the requirements and constraints, and the possible solutions.

Please comment and discuss inline.

Baseline
Goal: provide an MCR-capable interface for updating pages, with a minimum of refactoring of adjecant code. The implementation doesn't need to fully support MCR yet.


 * Use ContentHandler::getParserOutput to generate output for each slot
 * A true "baseline" doesn't require this. You only list it here because of the extra requirements you include below. Anomie (talk) 20:32, 26 February 2018 (UTC)
 * Use Content::getSecondaryDataUpdates to get data updates for each slot
 * A true "baseline" doesn't require this. This seems aimed at "SDC wants two slots blindly concatenated". Anomie (talk) 20:32, 26 February 2018 (UTC)
 * Update links tables with information from all slots. In the future, links tables may also record which slot references what resource.
 * A true "baseline" doesn't require this. This seems aimed at "SDC wants two slots blindly concatenated". Anomie (talk) 20:32, 26 February 2018 (UTC)
 * It's aimed at "SDC wants to be able to place the output of two slots freely on a page", along with "SDC wants to re-use the Wikidata representation and editing interface for the structured output", which I just confirmed as requirements with Mark and Ramsey. Plus "it seems really useful to be able to use output for different slots independently of each other for various use cases", which I confirmed with James. Do you disagree with these things even being requirements? -- Daniel Kinzler (WMDE) (talk) 18:32, 28 February 2018 (UTC)
 * Store a single ParserOutput for the revision in the ParserCache. Can be just the ParserOutput for the main slot for now, but will have to somehow contain all information from all ParserOutputs of all slots eventually. The code composing that combined output may be different for different "kinds" of pages, and may be defined by an extension. The mechanism for that doesn't need to be decided at this stage, though.
 * It won't necessarily have to contain all information from all slots. Even if it does contain some information from a slot, that need not be all the information. Trivial example: if we add a "metadata" slot to hold categories and such, there's no need for the main ParserOutput to contain whatever HTML might be generated to view that metadata. Anomie (talk) 20:32, 26 February 2018 (UTC)
 * If we want to have it in the ParserCache, it has to be in the PO. Or we re-design the ParserCache. -- Daniel Kinzler (WMDE) (talk) 18:32, 28 February 2018 (UTC)
 * The use of ApiStashEdit can be restricted to the main-slot-only case for now.
 * Only if EditPage is restricted to the main slot only. Anomie (talk) 20:32, 26 February 2018 (UTC)
 * No, it could just not use stashing if it's being used for multiple slots. This will be rare anyway, at least in the beginning. -- Daniel Kinzler (WMDE) (talk) 18:32, 28 February 2018 (UTC)

This implies that we need access to POs for individual slots and a combined PO during saving. This does not imply anything about the structure of the PO that gets written to the ParserCache or shown in a preview.


 * Anomie, Daniel Kinzler: just making sure that I understand what is the disagreement here: Brad would prefer an initial implementation in the spirit of T174039 (use the MCR interface, but completely ignore and/or error out on non-main slots), while Daniel would like to pull in enough functionality to make the SDC use-case work, right? The former seems like a more obvious initial goal, so is some time constraint the motivation to do more at once? Or are you worried that doing the change in small steps might lead into some architectural cul-de-sac? --Tgr (WMF) (talk) 10:20, 1 March 2018 (UTC)

Multi-Slot View
Model use case: Structured Data on Commons (SDoC). Single-slot viewing should also be possible.


 * Allow the rendering of one slots to depend on content of all slots. (Possible restriction: only allow the main slot to depend on other slots. This would allow the main slot to depend on the rendering of other slots, which could otherwise lead to circular dependencies). This needs ContentHandler::getParserOutput to have access to all slots.
 * Allow Article::view (or a replacement of that method) to show the content of each slot, rendered individually, in separate sections of the page. In particular, allow a rendering of the structured data to be shown that is independent of the wikitext, rendered in the user's interface language. That rendering also serves as the editing interface for structured data, just like on a Wikidata page.
 * This, IMO, should be a non-standard view. And it may not need to display all slots at once, versus showing individual slots. Anomie (talk) 20:32, 26 February 2018 (UTC)
 * I asked that question to the MultiMedia team. The result was that they want to at least have the option to have the standard view serve the full HTML of both slots. And it seems likely that this will be the case at least for the initial roll-out. -- Daniel Kinzler (WMDE) (talk) 18:39, 28 February 2018 (UTC)
 * What is meant by "standard" here? That no extension is customizing the rendering, or that the user is not customizing the request (so e.g. the request is for  and not  )? If it's used in the first sense, why is it important for the MM team what the standard behavior is? They are, after all, working on an extension to customize the rendering of file pages. --Tgr (WMF) (talk) 10:19, 1 March 2018 (UTC)
 * The code composing that combined output, when constructing and/or when using the PO, will be different for different "kinds" of pages, and may be defined by an extension. In the SDoC case, it will be specific to the File namespace, and will be supplied by the WikibaseMediaInfo extension.
 * This is an assumption that need not hold, and either limits the combining to a single extension per page or requires different extensions to hook into or extend each other. What if, for example, TimedMediaHandler wants a slot of its own for subtitles, with its own "combined output" code? Anomie (talk) 20:32, 26 February 2018 (UTC)
 * Since a completely generic combining mechanism seems insufficiently flexible (or would have to be extremely complex), and it's unclear how multiple extensions would cooperate on this, the present idea is to have a singe extension control the rendering of a page. I could also imagine us allowing a single extension to do complex composition, while allowing other extensions to add stuff at the top or bottom.
 * Anyway, what's the alternative? Am I understanding correctly that you want to use wikitext in the main slot as the only composition mechanism? -- Daniel Kinzler (WMDE) (talk) 18:39, 28 February 2018 (UTC)
 * Specifically in the case of TimedMediaHandler, it'll probably want to customize the MediaTransformOutput rather than the ParserOutput as the subtitles relate to the image and not the description wikitext. But in general, yeah, the ownership of composition seems tricky. One approach is that content handlers have access to what role the content is in (seems like a good idea in general) and can request to be included in the view. (So by default only the main content is visible but other slots can opt in.) Each of the visible slots could have access to all others and do its own composition then. --Tgr (WMF) (talk) 10:19, 1 March 2018 (UTC)
 * Note that for output in the user language, the ParserCache needs to be split (which is already the case for File pages on Commons)

Multi-Slot View (Anomie's alternative)

 * The default view will show the content of the main slot, as it does now for the non-MCR case.
 * The main slot can explicitly transclude content from other slots, via parser functions or tags, as can happen now. This probably won't generate a ParserOutput for that slot, although that's a possibility if the implementation of the function/tag finds that useful.
 * Content from other slots might also be implicitly "transcluded", much like how Extension:Cite does now when no is present in the wikitext. This need not generate a ParserOutput either, e.g. a "metadata" slot might just call addCategory, addDisplayTitle, and so on to add to the ParserOutput from the main slot. We could locate the code for implicit transclusion in hooks (as Extension:Cite's implicit  does now), or we could add it to Content or ContentHandler to avoid having to use hooks.
 * So, the main implication is that the default view will always be the main slot only. The only way to show things from other slots in the default view would be to explicitly access it from the main slot's wikitext (or other content), or by using hooks to do something very similar to what I am proposing with the combined PO.
 * Since pretty much all the use cases that have been discussed for MCR (Requests_for_comment/Multi-Content_Revisions) assume that the content of all slots is visible per default, it like a good idea to establish a standard mechanism for achieving this easily. I don't see how this can be done in Content or ContentHandler, I'd do it in a separate "PageOutputComposer" or "PageTypeHandler", since the composition logic doesn't have anything to do with the content model. Point in case, for e.g. template documentation, all slots would be wikitext. -- Daniel Kinzler (WMDE) (talk) 22:36, 28 February 2018 (UTC)
 * Of the use cases described there, structured media info, gadget styles and file history should be visible by default; infobox data, (probably) the page metadata, templatedata, templatestyles, (probably) video subtitles, assessments, focus area, CentralNotice bits, ProofreadPage and blame maps shouldn't. I could see categories and template/module docs go either way - get their own "view slot" or just get pulled in by the main content. (Arguably they should work with arbitrary types of main content so maybe getting their own slot makes more sense.) Some use cases discussed elsewhere were Wikidata description overrides and page annotations, which also should not be visible. So only displaying the main content seems like a reasonable default. Having something like  seems like an OK way to allow easy opt-in, although the content handler would have to know the role for that (and maybe at that point it makes more sense to have some kind of role handler?). As for PageTypeHandler, how would it differ from the content handler of the main slot? We have no such thing as page type. --Tgr (WMF) (talk) 11:13, 1 March 2018 (UTC)
 * It should also be possible to view an HTML rendering of a non-main slot. This would generate a ParserOutput for the slot.
 * The parser cache will likely need to be extended to be able to cache non-main slot ParserOutput objects, both for the non-main viewing and for use by potential implicit transclusion. Whether we cache only the main slot's pre-implicit-transclusion ParserOutput (and run implicit transclusion as a post-cache transformation), only the post-implicit-transclusion ParserOutput (and thus re-parse the main slot whenever any slot changes), or both is yet to be determined.
 * I was trying to avoid that. Partially just to avoid additional work, but also because code reading from the ParserCache expects the PO in there to be "the page output". Which it wouldn't be if we only put the main slot there. Basically, my thinking is that all slots contribute to the page's output. Your approach seems to be that the main slot is the page output, but there may be other page output, which may or may not be tagged onto the main output. -- Daniel Kinzler (WMDE) (talk) 22:36, 28 February 2018 (UTC)
 * Anomie, I just realized: your pre-implicit-transclusion output is my "main slot" output. Your post-implicit-transclusion output is my "combined" output. We are not disagreeing here. It's the same thing. We need both somewhere - and working on the initial WikiPage refactoring, I found that things become a lot easier if both are available from the PreparedEdit (I was trying to avoid that, but kept running into trouble). We can keep talking about how the code that does the "implicit transclusion" aka "output composition" lives - but that discussion does not need to block the current patch. For that patch, the only thing that matters is that we need a way to a) get a PO for each slot and b) get a PO with implicit transclusion (aka output composition) applied. And I did not find a way to avoid having both in PreparedEdit -- Daniel Kinzler (WMDE) (talk) 13:49, 1 March 2018 (UTC)
 * Access to other slots of the revision will proceed by loading the revision, normally by using Parser::fetchCurrentRevisionOfTitle. If a Parser instance isn't available, ParserOptions::getCurrentRevisionCallback could be used more directly or via a similar helper.
 * This could work, but we'll want to be able to use this before a Revision has been saved. In my mind it makes more sense to have a way to pas the ParserOptions for other slots - which may or may not come from the revision. -- Daniel Kinzler (WMDE) (talk) 13:49, 1 March 2018 (UTC)

The SDoC case could work like Extension:Cite's : if the wikitext includes a tag to say "insert the structured data here" it will do so, and if not it'll append the structured output to the main slot's wikitext.
 * Appending to the wikitext will not work, it would need to append full HTML, plus registering the necessary JS modules, CSS modules, tracking info for the links table, etc. This sounds a lot like the "combined" PO I was suggesting, but it's unclear to me how the Wikibase content can register its DataUpdates. Probably by hooking in again at some later point. Basically, this sounds like an emulation of my proposal using hooks instead of defining a mechanism for composing a PO from multiple slots.
 * Overall, this seems to be more work: we have to modify the parser cache, introduce new hooks, provide ways to access all kinds of content from withing wikitext, or use several hooks for every slot so it can attach itself to the main PO. I'd rather have a mechanism that just defines how the combine PO is composed, with a per-page-type handler for combining the HTML in the desired way. -- Daniel Kinzler (WMDE) (talk) 22:22, 28 February 2018 (UTC)


 * I think there are orthogonal issues here:
 * Pull model (only the main content is parsed; it can initiate the parsing of other slots if it desires) vs. push model (all slots are are parsed and then the ParserOutput objects merged somehow). The latter seems easier to coordinate.
 * PO object hierarchy: should the non-main content POs be merged into / provided to the main content PO, or should all the content POs combined into a new revision-level PO? I guess that depends on whether we expect that typically the main content type determines how combining should work, or it's something that would typically be controlled by an extension that registers itself in some content-type-unrelated way. I can't really think of any counterexamples to the first option.
 * --Tgr (WMF) (talk) 11:13, 1 March 2018 (UTC)

Multi-Slot Editing
Model use case: Atomic editing of a Lua module and its documentation. Single-slot editing should also be possible.


 * Allow EditPage to show one editing area (for text based content: one text area) for each slot. Also provide a single-slot editing mode.
 * Allow previews to be rendered for all slots or individual slots.
 * When parsing/rendering after save, re-use the cached rendering of unchanged slots, instead of regenerating it needlessly. However, if the rendering of the unchanged slot depends on a slot that was touched, it needs to be re-rendered anyway.

This implies that ApiStashEdit, ApiParse, and ApiEditPage all accept input for multiple slots at once.


 * IMO trying to figure out the edit view based solely on architecture requirements and only incuding designers when it's fully implemented will end badly. Just throwing out a bunch of text boxes under each other is probably not acceptable UX. (It will be needed as a non-JS fallback as there is little else to do int that case, but usually some mechanism will be required to decide which component gets to control the rendering. Maybe some kind of tabbed view would make sense for handling multiple large edit boxes. TimedMediaHandler might have 200 slots for subtitle translations that it wants to handle in a single view block, without interfering how the rest of the content is rendered. VisualEditor takes over the whole content area, it might want to force the non-main components into dialog boxes. What would be a reasonable interface on a mobile screen? etc.) It would be good to ask for a set of mock designs based on the use cases collected so far, just to have some ideas to base the architecture planning on. --Tgr (WMF) (talk) 07:25, 1 March 2018 (UTC)

Proposed Changes to ParserOutput

 * Each ParserOutput can record which slots it depends on
 * The combined canonical PO has to return aggregated information of all slots from all getters
 * The combined canonical PO should not contain duplicate information, since it will be serialized and stored in the parser cache, so size matters.

Option 1, compose output just in time:
 * The combined ParserOutput for all slots would have to
 * know the ParserOutput objects for each slot
 * getters aggregate info from all the POs on the fly.
 * know how to combine the HTML from all slot's ParserOutputs into a single output (or return null from getText), by substituting placeholders in an HTML "template". Note that this template may be defined or modified by an extension, and may depend on page type or namespace or some other property of the page.

Options 2, compose output right away:
 * The combined parser output gets fed with the aggregated content of the POs.
 * The code that constructs the combined PO also constructs the combined HTML right away.
 * This means that during saving, the POs of unchanged slots cannot be re-used.

Note that option 1 and 2 offer largely the same interface to calling code. The only code that needs to be aware of this distinction is the PO itself, and the code that constructs it.

This means that we can start with option 2, and change to option 1 later when need arises, for optimizing the rendering during save, or for other purposes. E.g. AbuseFilter may want to apply different rules to different slots, so it would have to process the POs of the individual slots separately.


 * Since all this depends on your model of how things work that I disagree with, I find all this unnecessary as well. Anomie (talk) 20:32, 26 February 2018 (UTC)
 * If I understand correctly, you want to "somehow" merge all info from the extra slots into the ParserOutput of the main slot. That seems very close to my proposal, the only difference being whether we conceptually have a "combined" PO that is different from the main slot's PO or not. I think it would be useful for this discussion if you could state your proposal below, and explain how it meets the requirements above. If you disagree with the requirements, please explain how and why. -- Daniel Kinzler (WMDE) (talk) 18:27, 28 February 2018 (UTC)
 * Actually, for the "combine output using hooks" version of your proposal, the need/opportunity for optimization is the same. The difference is then just whether we have separate keys for each slot in the ParserOutput. I like separate keys in principle, but see little practical advantage, and I expect sticking with a single key is going to be a lot simpler. -- Daniel Kinzler (WMDE) (talk) 23:05, 28 February 2018 (UTC)