User talk:Daniel Kinzler (WMDE)/MCR-PO

I'm not convinced that the assumption that visible slots should just be concatenated to the primary one and handed to the skin as a single string (as opposed to handing an array with separate PO outputs for each visible slot) is a good one to have. That would make it the ParserOutput's responsibility to decide how to stitch them together, but that really should be a skin-level decision. (E.g. a simpler skin might want to just display the slots below each other, a more complex one might want to use some kind of selector-based interface such as tabbed navigation.)

On the other hand, should skins know about revision slots at all? The point of MCR as I understand it isn't to have subpages of actual content within one page where a tabbed interface might make any sense. It's to have multiple Content objects that are combined to build one page rather than having to do things like magic subpages (e.g. Scribunto doc pages) or embedding JSON in the wikitext (e.g. TemplateData).

I think those are both points of MCR. Sometimes you'd want content objects to combine before the skin gets involved (e.g. TimedText probably needs to integrate with the main content in some fairly specific way and putting it behind a tab makes zero sense), sometimes the secondary content is fairly generic and can be used with pretty much any main content type (documentation, or Wikibase metadata) in which case it's just a second thing on the page and skins can get creative in finding the best way to handle that (you probably wouldn't want to handle things like mobile vs. desktop layouts on a content handler level).

There are even types of content which some skins might want to handle in a special way (e.g. Timeless puts categories into the sidebar, Winter wanted to put infoboxes there) - MCR is a way to coordinate that. (I'm not sure it's a good way - but at least worth considering.)

Neither categories nor infoboxes are good cases for a skin needing to know about MCR, IMO. Categories are already separated better at the ParserOutput level. Infobox pulling I'd hope could be done with a CSS class so it can be responsive to screen width, rather than server-side skin code.

Yeah, for categories (and IW links etc.) handling them via the PO is probably better. For infoboxes, I don't think CSS is a good option. The way Winter wanted to handle them, I can't even begin to think how you'd do that with CSS. And Minerva doesn't want them in the HTML at all, to cut down on size. So there should be some way for skins to "pull out" content slots - tell the SlotCombiner or primary RoleHandler or whatever ends up controlling the main content HTML composition to omit that content slot, and let the skin handle it directly.

Maybe that should happen on some different level than MCR (e.g. the infobox would be rendered with {{#infobox}}, and the parser function callback would provide a hook that the skin can interact with, and just return an empty string to the parser when the hook is used). But we should keep the option open IMO.

If a skin needed to pull out the HTML for infoboxes on the server side, I'd probably recommend it be implemented in the skin as a DOM manipulation on the output HTML (e.g. using HtmlFormatter). We probably don't want to fragment the parser cache on skin for all the articles using infoboxes.

Rendering to HTML and then re-parsing it is not a sane approach: it's fragile, resource-intensive and loses a lot of context. Fair point about cache fragmentation on the other hand.

In the long term we are moving towards caching pages in smaller shards (so that a template edit does not result in surrounding parts of the page being invalidated); this kind of functionality could work on top of that (ie. have some kind of placeholder saying this is the place of the infobox, and then decide whether it gets replaced with the infobox HTML or the empty string on some kind of cheap cache shard composition operation instead of an expensive parse. But that's a big change and tying it together with MCR seems like a really bad idea.

Who said anything about reparsing? I was talking about extracting the HTML of the infobox whole.

I've heard talk about that sort of partial page update for years (it was a selling point of Parsoid when it was first being developed, for example), but I haven't seen anyone actually doing it.

Parsing as in e.g. loading it into a DOMDocument, I mean.

Resource-intensive, maybe. But I can't see any way in which it would be fragile or lose context beyond position in the page, which you'd have lost anyway with your proposal of pulling it out inside the Parser.

This is a tangent to the main thread .. but incremental parsing is still on the charts, and we had a limited form of it in Parsoid for a while, which we have disabled for the last couple years because of some unprioritized issues we had to fix. But, it is hard to reap the full benefits of it while templates can do arbitrary things in the general case. Hence we just decided to look at the general problem once the balanced / typed templates work is further along.

@Daniel any thoughts about this? Your proposal is less clear on this point but also seems to assume that callers interact with a single output (such as a combined ParserOutput), and if e.g. wikitext and mediainfo both need to be visible in the default view then concatenating the outputs of those into a single HTML string would happen somewhere in the MCR internals (e.g. done by the PageTypeHandler).

I don't think that's a good assumption to have. Most of the time non-main slots won't be directly visible, but when they are, how to combine them is a UI concern, which means it should happen in the view layer and should be customizable by skins. There should be a way to get the combined output as a single HTML blob for backwards compatibility, but there should be a way to get the visible slots separately and it should be assumed that skins might do that and make their own decisions about what to do with them. If that happens via a combined ParserOutput that has some extra method to expose the visible slots, that's as good a method as any, but this expectation should be part of the requirements IMO.

One somewhat tangential but related thought: I think part of the confusion stems from the fact that MCR is an implementation of some abstraction at the wiki level. It would be useful to articulate what that abstraction is.

Or, is the concept of { page, slot1, .. slotn } just a low-level platform "api" for applications / products to implement whatever abstractions they need for their application?

For example, if MCR is an implementation mechanism for a single unified abstraction (vs being a low-level platform "api" for products), then what extensions, skins, APIs, gadgets get exposed to is this abstraction, not a page + slots or main slot + auxilliary slots, etc.

However, if MCR is low-level platform api for products, then, the products that build on top of this platform are responsible for exposing their specific abstractions. You would still not expose the slots api.

Would answering this more fundamental question help clarify the differences and debates going on right now? Or, am I somehow missing the picture here?

The basic idea, as I understand it, is that we want to be able to have a wikipage that's made up of multiple types of data, "versioned" together instead of being separate pages but not having non-wikitext data embedded in wikitext like TemplateData does.

For example, SDC wants a file page to have both wikitext and Wikidata-style structured data (and the file might be considered a third slot). A template might want to have the wikitext, and TemplateData JSON not embedded in the wikitext, and a TemplateStyles stylesheet (and, depending on how protection and reparsing after edits winds up working, maybe documentation wikitext too). A gadget might want wikitext, JavaScript, and CSS.

I'm not sure how that relates to your questions.

Yup, I understand that concept of slots.

Right now, there is the Everything is a wiki page abstraction which everything builds off of and which is what is exposed through the APIs, etc. I think / assume that abstraction will continue to hold even with the introduction of MCR. For b/c reasons, I assume this page will be the page (or main slot) in the post-MCR world.

However, the question is if there is a different (additional) abstraction that is now exposed by MCR. If so, is { page, slot_1, .. .slot_n } that abstraction that is exposed independent of what applications do with it? Or, does every application (say SDC, TemplateStyles, Gadget) expose its own app-specific abstraction via its app-specific API endpoints. I was making the observation that answering this question might perhaps help clarify what the implementation ought to do, i.e. should / does the skin manipulate its own slots (yes for page+slots, depends for the app-specific abstraction).

I think "everything is a wiki page" continues to hold. It's just that a wikipage revision might now have more than just one Content object.

Gadget is at one end of a spectrum: it defines a new thing ("gadgets") and will want to control everything about that thing. TemplateStyles is at the other: it wants to add just one slot to extend existing templates, without trying to control the whole idea of "templates". SDC you could think about both ways, either controlling the whole idea of "file descriptions" or adding on to file descriptions.

TemplateStyles probably won't do anything really new for MCR, it could just let the tag pull from the non-main slot if one exists rather than the main slot and let generic contentmodel-based editing code handle the editing of the slot. Gadgets too, it just would have to pull from non-main slots instead of separate .js and .css pages. I don't know whether SDC would use the same APIs that are used on Wikidata or if it'll have to do something new.

If we actually wanted a skin (as skins currently exist) to do something useful with slots, we'd have to somehow have OutputPage transfer the relevant data from ParserOutput to the skin, whatever "the relevant data" turns out to be (i.e. we'd probably have to extend ParserOutput to hold "the relevant data"). At the moment I think we're well into YAGNI territory on that.

"Everything is a wiki page" really says that everything is content (ie. has a certain set of transparency and moderation related behaviors associated with it, like history, diff, revert, protection, deletion, suppression, checkuser etc) right? Those behaviors should still exist but some of them will be interpreted on the slot level, or both the slot and page level. Exactly which is still an open question (edits and consequently checkuser and revert will still be page level; deletion/suppression will be per-page; diffs, APIs and export will work on the slot level, with some kind of B/C mechanism; how to handle protection and AbuseFilter-like tools is less obvious).

Which is to say, yes, it would be helpful to spell out what the abstraction is (what should be considered a page-level behavior and what a content slot level behavior). Is that what you meant, @SSastry (WMF)?

At the moment I think we're well into YAGNI territory on that.

How do we make sure that's the case? When designing an interface, "just extend it later when the need arises" is not a great approach.

Mine was a more general question. Let us take a page. Do different applications get to add their own app-specific slots to a page? If so, what is a revision now? Is it the page + all slots defined by all applications?

One way to clarify what I am asking is to contrast this with earlier proposal for pages which was mime-based multi-content pages. In that proposal, it is very clear that a page = page + a common set of meta-data blobs stored in mime-slots on the page. So, there is still a unified abstraction of a page (it just happens to have a mime layer on top) independent of the application that operates on a page.

But, from all I have gathered so far from reading about MCR, and to answer my own earlier question, it appears that MCR is a low-level platform API, and it is upto higher level layers to provide the necessary abstractions and conceptual consistency. So, it seems to me that you (not individual you, but generic you :)) need to pick an application, like say SDC, and work through the implications of SDC providing the abstractions and consistency and what the requirements will be of the underlying MCR platform API.

The revision is the state of all the slots that exist in that revision, yes. But the rest is something Daniel and I have been disagreeing on. Daniel's model (as I understand it) rests on the assumption that one extension controls the page and all its slots, so multiple extensions would have to be intimately aware of each other and more or less implement a superextension that's the combination of the two when both need to affect the same page. The model I propose lets each extension control its own slot, so multiple extensions can coexist with few concerns beyond maybe ordering.

I don't see much difference between the "mime-based multi-content pages" and MCR, although this is the first I've heard of the former. Based on its name the former would presumably have one multipart/X content blob holding the individual data content blobs, while MCR does the same by organizing the data-blobs with slots and content DB tables instead. Then we get to the disagreements: do we have "multipart/foo", "multipart/bar", and so on with a handler for each that interprets the data content blobs (Daniel), or do we have a generic "multipart/mw-content" and determine what to do for each data-blob based on its role metadata (me)?

SDC, at the initial stage at least, wants two data-blobs: one with the existing wikitext and one with Wikibase-style structured data. It wants the standard page view to include HTML from both blobs (and with all the fancy JS that wikidata does for its editing on the wikibase blob).

At the low level we just need code that stores the two blobs and lets them be read back. That part isn't controversial.

What we're arguing about here is the model for turning those two blobs into HTML (and updates for links tables and such): a dedicated SDC PageHandler that combines the two, or generic code that parses the main-slot wikitext and then tells the SDC SlotHandler to combine the SDC slot's data into that? Then there are/were some additional disagreements, such as whether the PageHandler/SlotHandler works with the slots' Content objects or ParserOutput objects generated (as if standalone) from those Contents, whether ParserOutput caching would remain keyed on revision ID only (with per-slot POs somehow embedded inside the one cached PO) or should be extend to be able to key on revision ID + slot role ID for per-slot POs (and still using plain revision ID as the cache key for the combined PO), etc.

"diffs, APIs and export will work on the slot level" seems like oversimplification to me. Diffs, sure, although I believe the current plan is that the UI diff actions will still take revisions as input and will concatenate the diffs for the different slots. APIs depend on the API, but for the action API the plans I have in phab:T174032 are largely remaining on the page/revision level with additional properties to select the slots where applicable, no new slot-level API modules. Export is probably going to be page- or revision-level too, just with each revision having multiple content blobs rather than just one.

AbuseFilter works on the level of an edit producing a new revision, although what that will look like in the filter language for an MCR edit is yet to be determined. At least to start, protection will likely remain page-level.

The point of YANGI is that if you try to guess what you're going to need without any real use cases then you're likely to guess wrong and have to extend the interface anyway. Especially since we already have OutputPage between ParserOutput and skins, I don't see any need to worry about skins and slots now.

There has been some back-and-forth on this, so maybe it's worth to go through the use cases in detail.

• Main wikitext: should be visible, but it's the main slot. (The long-term structured data UI plans would actually hide it behind a tab, but I imagine we'd want to do that with Javascript while still keeping the content in the default HTML view.)
• Structured media info: should be visible.
• Infoboxes and similar "page widgets": typically, the infobox should be rendered inside the article, and not as a separate content block, so as far as MCR is concerned it should be invisible. There might be exceptions (e.g. Winter-style skins) but there it would get some kind of special handling.
• Categories: should be invisible, as categories are handled in a different way, not via ParserOutput, and we probably don't want to mess with that.
• Other metadata: in the case of interwiki links: same as categories. (Also these usually come from Wikidata so cannot be handled on an MCR level.) Other types which aren't visible currently (NOINDEX etc): should be invisible, there is nothing to show.
• Documentation subpages (template + module): should be visible, this is really the main content of the page as far as viewers of the template page are concerned.
• Template styles: should not be visible by default, it's similar in function to the template source code (which we also don't show).
• Video subtitles: unclear. If we put subtitle translations in their own slot, those definitely shouldn't be visible. Chances are subtitles should be rendered by TimedMediaHandler on top of the video and not be visible as far as MCR, Article::view and other generic code is concerned. OTOH some video players put the subtitles below the video (e.g. this older look for YouTube), for navigation and for the benefit of search engines, so that would be an option (same for audio transcripts).
• Page collections: this doesn't seem like something that should be handled by MCR, actually.
• Article assessments: unclear. Could keep the current template-based display, and just make the way the data is stored more standard; or the assessment box could be generated by the extension in which case it would probably be displayed as a separate slot.
• Image focus areas: should not be visible, this is not something that can be meaningfully shown to the reader. Even if we store the image + focus area together as a slot, the way the image should be shown needs to be deeply integrated with the skin, so it has to be hidden as a slot and then rendered by some other mechanism. If the skin doesn't support article banners, it shouldn't be displayed at all.
• MassMessage target lists: these will be main content slots.
• CentralNotice CSS/JS/HTML/wikitext fragments: presumably the default view we want here is to show the banner as it will look in the article, and not the various pieces of source code? But could see this going either way.
• ProofreadPage: unclear. Currently this just concatenates header, body and footer before parsing wikitext, not sure of the pros/cons of keeping that vs. displaying each slot separately. Anyone familiar with ProofreadPage usage patterns?
• Gadget files: unclear. Could show everything on the same page, or just show the gadget definition and link to the individual slots, to make the page size more manageable (imagine a gadget that pulls in a 30K external library, you'd probably want to look at the locally authored files without having to scroll through that).
• Workflow state: probably should be visible.
• Wikidata description override: should not be visible. (Wouldn't make sense since the description isn't either. Even that does get displayed, that needs some non-slot-based mechanism as the text can come from Wikidata.)
• Annotations: these need highly specialized display integrated with the main content HTML, so should not be visible as slots.
• Page forms: should be visible.
• Blame maps: should not be visible (not even as hidden-but-included-in-HTML as these tend to be huge).
• OCR text: should be visible.
• Files: should be visible, but probably it would be the main content slot for file pages if we do this refactoring.
• Rendered HTML: does not seem terribly likely that this will ever happen as part of MCR.

So that's 5 use cases for non-main slots which should be visible by default, 8 which shouldn't, 5 in which case either approach seems reasonable.

Does that seem like an OK tally?

Page forms could be in the same situation as infoboxes.

I doubt it's that useful to put the OCR text of a large PDF or the like into the default HTML rendering. This seems like a good candidate for having a link to view it rather than just merging the raw output of the slot.

And I hope TimedMediaHandler doesn't actually add a slot per language. Anomie (talk) 17:25, 1 March 2018 (UTC)

What would be the alternative there? We want to be able to filter by language in the page history, I imagine, so we can't just put everything in one slot. Same goes for template subpages (documentation, maybe even localization) on multilingual wikis.

This is somewhat relevant because if there are pages which have hundreds of slots do to i18n then always generating a ParserOutput and SecondaryDataUpdates for every slot seems like a no-go.

One item of note is that role_id is a smallint, so one extension using up 400-some of the 32767 available IDs could be problematic. Especially if it becomes a pattern.

If it becomes a pattern, maybe the slot table should have an optional language field?

I18n as a first-class citizen is a core principle of MediaWiki and I don't see how that could happen without some sort of support for slot localization.

TimedMediaHandler might use a format that can store multiple languages in one slot, as Wikibase does (I believe) for its translations of data strings. Or TMH could keep using its own namespace with subpages.

Or maybe a "slot_lang_id" could make sense. But it's not the only option, and could have drawbacks such as making the history of the "main" version harder to follow with all the updates to translations.

I don't think those are great alternatives. Putting everything in a single slot would make following the main version (or any other) even harder; using subpages preserves all the problems that MCR was supposed to solve (e.g. things like export, protection, move not working out of the box).

As discussed on the subject page, a baseline version of the patch would completely ignore non-main slots (or require that there aren't any). Wouldn't the best short-term path forward be to rewrite the patch to do that? That would unblock further work, make it possible to roll out a smaller set of changes at a time (although I guess it makes little difference in practice as long as it's impossible to create non-main slots) and the content of that patch does not seem controversial.