User:SSastry (WMF)/Notes/Document Composability

WIP: This is a pulling together of notes from separate documents (Parsoid/DOM notes, Wikitext notes, Notes about incremental parsing) and from different phabricator tasks into a coherent writeup.

Composability
The main requirement is one of being able to compose a document from fragments, independent of how the fragments are generated (plain markup, transclusions, extensions, widgets, components, etc.)

Given document composability, So, it is sufficient to focus on this high level goal of enabling composability of a document from document fragments.
 * it is simpler to reason about the document and the markup
 * WYSIWYG editability is enabled more easily
 * edits can be processed lot more efficiently
 * the target document can be easily tailored to different device / network contexts
 * ... other benefits ...

Questions

 * 1) Do we need the ability to compose document fragments? If the answer is no, we can pack up and go home.
 * 2) What are good candidates for document fragments?
 * 3) * 2a. How are they specified in input markup? Are they implicit or explicit? If explicit, are they opt-in or opt-out markers?
 * 4) * 2b. How are they identified in output markup, if at all?
 * 5) What kind of constraints need to be respected while composing a document from fragments?
 * 6) What approaches exist for enforcing constraints?
 * 7) How are old revisions handled?
 * 8) What pieces of this functionality need to be supported in the core parser and Parsoid?

Wikitext and composability
Wikitext does not have the notion of being able to compose the final document from individual document fragments. Even with transclusions and extensions, what is composed is the source markup not the target document, i.e. you construct the complete / final source and derive the target document from it.

What are good document fragments?
If we want to support document composition from fragments, we need to figure out what are good candidates for documents fragments and how they are specified? A related question is: do these fragments need to be marked up specially in output?
 * T114444 proposes that sections, lists, tables, can all be treated as document fragments. The RFC proposes that these boundaries are specified by changing wikitext semantics (DOM scopes). So, there is no explicit markup or opting in required.
 * T114445 proposes that templates be treated as fragment generators. Specifically, it proposes that template authors opt in to this behavior by adding markup to template output.
 * T114072 proposes that sections be treated as document fragments (and specifically requests special output markup). There are some details to be worked out as to whether this behavior is enforceable everywhere or not.
 * T105845 is somewhat related. It picks a narrow subset of templates used for navboxes and infoboxes and discusses other DOM-generating components. The primary concern is about the form of the output DOM and requests special output markup as with T114072.
 * T114432 is tangentially related. It proposes heredoc syntax for templates for the multi-template-content-block use cases. The new syntax effectively demarcates document fragments as well even if that is not the direct intention.

How are document fragments identified during parse?
There are multiple approaches here. Which of the above strategies are suitable depend on requirements of the core parser, Parsoid, what are the needs of specific clients / products / projects on the Wikimedia cluster, what kind of support we want to provide for 3rd party wikis, how old content is affected. This needs additional clarity.
 * Specified via wikitext semantics: For example, at some point, we can declare that lists and tables produce well-formed DOMs and are processed as such. This is unlikely to work for transclusions as of today.
 * Opt-in via special markup: T114445 specifies parser-function style markup added to template source allowing template authors to declare how the output of that template should be processed. The opt-in / typing information can also be added to templatedata.
 * Opt-out via special markup: This is based on the optimistic assumption that most templates do indeed produce well-formed output, but there might be scenarios where this is not possible (ex: multi-template-content-block use cases, templates used for generating strings to be processed as HTML attribute strings or HTML attribute values). The opt-out information can also be added to templatedata.
 * Automatically inferred: This is a special case of opt-out/opt-in where template behavior and fragment boundaries are inferred based on information available during regular parses. Conceivably, this derived information could be added to templatedata. So, the difference from opt-in/opt-out is whether a template author adds this information or whether software adds this information.

How are document fragments identified in the output document?
Parsoid uses special markup for transclusions and extensions. T114072 and T105845 propose / request special markup for other document fragments. But, looks like this special markup for document fragments is a cross-cutting concern. A parser might add additional information (ex: Parsoid in the private data-parsoid attribute) to aid incremental parsing. But, in all cases, it makes sense to come up with output markers that delineate document fragments and provide additional information about how they were generated, etc.

Composition constraints
Composability, to be useful, has to ensure that the final produced document is always semantically meaningful.

If there were no composition constraints, this discussion would be vastly simpler. All that would need to be done is demarcate fragment boundaries (ex: sections, templates, extensions, lists, tables, etc.), parse those to DOM to fix ill-formed markup and compose the final document from top-level output by inserting output of individual fragments at the right places.

But, there are two kinds of contraints that apply [ Note that this distinction is probably mostly pedantic. We are, at this point, constructing HTML5 documents as our canonical target (and deriving all other formats: text, PDF, mobile app output, audio readers, etc.) from this canonical form. ]

1. Composition constraints that are independent of the target domain: For example, it doesn't make any sense to nest a link within a link, a paragraph within a paragraph, or a heading within a heading.

2. Composition constraints that are imposed by the target domain (HTML5): While HTML5 specifics content model constraints, for the most part, the constraints that are actually enforced are ones that fall out of the tree building algorithm that parses a (html) string into a HTML5 DOM.

For example, see the following output: So, within HTML5, you cannot nest a list within a paragraph. You cannot insert content in a table anywhere except within ,  , and tags. Content inserted anywhere else within a table gets moved out of the table (the content is adopted by the 's preceding sibling. There are likely other such constraints as well.

Possible approaches for handling composition constraints
a. No-op: This is the ideal scenario where nothing needs to be done because no constraints are violated.

b. Modify fragment: Based on the insertion site, modify the output of the fragment suitably to ensure that constraints are respected. For example, if a fragment is being inserted inside a link, you could convert all links in the fragment to plain text.

c. Change insertion context: So, if a link-containing fragment is being inserted inside a link, you could insert the fragment *after* the link.

d. Expand the fragment scope: So, when you try to insert a list within a tag, the p-tag is treated as the composable fragment instead of the list. This is how Parsoid deals with transclusions today. It has a notion of "template-affected output" which includes a template's output as well as any enclosing page context that is treated as an indivisible unit for the purpose of editability. Example: Look at Parsoid's output for a\n \nb 

e. Turn off fragment composition: Recognizing when document composibility cannot be supported and mark the document as such. In this scenario, WYSIWYG editing and incremental parsing would be impaired or disabled on that page. Example: If you try to insert a list within a ... b   , fragment modification would have to convert the list into plain text which might be considered unacceptable. Similarly, changing insertion context to insert the list *after* the p-tag could be considered unacceptable in some contexts. So, if strategy d above is not available, another option would be to lose fragment composition ability on this part of the document. When the fragment is inserted in the paragraph, there are non-local effects on the document and the p and b-tags are modified as well. So, on any edit to this part, you would have to reparse the entire page. Similarly, inside say VE, you may get surprising non-WYSIWYG behaviour. Replacing the list-production-transclusion with plain-text might show you 3 paragraphs in VE, but on save, the content of those paragraphs will render as a single paragraph.

If we want to treat transclusion output as document fragments (which is the direction we seem to be moving towards), my personal instinct is that no single strategy is likely to cover the diverse templating use cases on the wikimedia wikis. The choice of what strategy is used / applied is also likely going to be determined by what the core parser needs to and can support, what Parsoid will support, and what the needs are for 3rd party wikis.

Situation today in core parser and Parsoid
As it stands today, in the core parser, strategy e. is applied everywhere implicitly even when it is probably true that strategy a. can be used commonly in practice.

Parsoid already supports some ad-hoc document composition via fragments. See below for why it is ad hoc.
 * It treats extension output, link text, and image thumbnails as document fragments. It does this to contain the effects of non-well-formed markup to those fragments. Parsoid uses fragment proxies to deal with composition constraints -- in other words, strategy e. is applied everywhere implicitly since strategies b, c, d aren't exercised.
 * Parsoid doesn't treat transclusions as fragments exactly. However, since Parsoid's original target was VE, it applies strategy d. to all transclusions to minimize surprising non-WYSIWYG behavior. So, in other words, Parsoid does treat transclusions as document fragments, but only for the purposes of visual editing. The information in data-mw can be used by clients to identify transclusions that behave like fragments. Specifically if data-mw.parts.length === 1, the transclusion does indeed behave like a composable fragment.

Given Parsoid's ad-hoc and incomplete handling of document composition, it makes sense to approach this more systematically. This requires moving wikitext semantics to enable document fragments and composition.

Composition constraints for children of
For top-level (children of ) document fragments, composition is fairly trivial. You can just drop in the fragment in all cases except for /   /  fragments which only make sense in a context. For such fragments, strategy b. of modifying the fragment to wrap a around the fragment is sufficient.

Composition constraints for transclusions
The focus of T114445 is to treat templates as fragment generators. Since transclusions can show up anywhere, constraint handling is trickier. My personal opinion is that all the non-no-op strategies should be on the table. I don't think any single strategy by itself is sufficient to cover all use cases.

T114445 attempts to tackle this problem as follows. Where composition is desired, it requires the template author to declare  a type for the template's output. The current prototype attempts to integrate the HTML5 tree builder into the core parser and let things shake out how they will. In this solution, the core parser won't suppport incremental parsing, but the goal is for Parsoid and the core parser's rendering to be identical.

... to be continued ....