User:SSastry (WMF)/Notes/Document Composability

WIP: This is a pulling together of notes from separate documents (Parsoid/DOM notes, Wikitext notes, Notes about incremental parsing) and from different phabricator tasks into a coherent writeup.

Composability
The main requirement is one of being able to compose a document from DOM fragments, independent of how the fragments are generated (plain markup, transclusions, extensions, widgets, components, etc.)

Given document composability, So, it is sufficient to focus on this high level goal of enabling composability of a document from DOM fragments.
 * it is simpler to reason about the document and the markup
 * WYSIWYG editability is enabled more easily
 * edits can be processed lot more efficiently
 * the target document can be easily tailored to different device / network contexts
 * ... other benefits ...

Questions

 * 1) Do we need the ability to compose DOM fragments? If the answer is no, we can pack up and go home.
 * 2) What are good candidates for DOM fragments?
 * 3) * 2a. How are they specified in input markup? Are they implicit or explicit? If explicit, are they opt-in or opt-out markers?
 * 4) * 2b. How are they identified in output markup, if at all?
 * 5) What kind of constraints need to be respected while composing a document from DOM fragments?
 * 6) What approaches exist for enforcing constraints?
 * 7) How are old revisions handled?
 * 8) What pieces of this functionality need to be supported in the core parser and Parsoid?

Wikitext and composability
Wikitext does not have the notion of being able to compose the final document from individual DOM fragments. Even with transclusions and extensions, what is composed is the source markup not the target document, i.e. you construct the complete / final source and derive the target document from it.

What are good DOM fragments?
If we want to support document composition from DOM fragments, we need to figure out what are good candidates for fragments and how they are specified. A related question is: do these fragments need to be marked up specially in output?
 * suggests that sections, lists, tables, and potentially other constructs can all be treated as DOM fragments. The RFC proposes that these boundaries are specified by changing wikitext semantics (DOM scopes). So, there is no explicit markup or opting in required.
 * proposes that templates be treated as fragment generators. Specifically, it proposes that template authors opt in to this behavior by adding markup to template output.
 * proposes that sections be treated as DOM fragments (and specifically requests special output markup). There are some details to be worked out as to whether this behavior is enforceable everywhere or not.
 * is somewhat related. It picks a narrow subset of templates used for navboxes and infoboxes and discusses other DOM-generating components. The primary concern is about the form of the output DOM and requests special output markup as with.
 * is tangentially related. It proposes heredoc syntax for templates for the multi-template-content-block use cases. The new syntax effectively demarcates DOM fragments as well even if that is not the direct intention.

How are DOM fragments identified during parse?
There are multiple approaches here. Which of the above strategies are suitable depend on requirements of the core parser, Parsoid, what are the needs of specific clients / products / projects on the Wikimedia cluster, what kind of support we want to provide for 3rd party wikis, how old content is affected. This needs additional clarity.
 * Specified via wikitext semantics: For example, at some point, we can declare that lists and tables produce well-formed DOMs and are processed as such. This is unlikely to work for transclusions as of today.
 * Opt-in via special markup: specifies parser-function style markup added to template source allowing template authors to declare how the output of that template should be processed. The opt-in / typing information can also be added to templatedata.
 * Opt-out via special markup: This is based on the optimistic assumption that most templates do indeed produce well-formed output, but there might be scenarios where this is not possible (ex: multi-template-content-block use cases, templates used for generating strings to be processed as HTML attribute strings or HTML attribute values). The opt-out information can also be added to templatedata.
 * Automatically inferred: This is a special case of opt-out/opt-in where template behavior and fragment boundaries are inferred based on information available during regular parses. Conceivably, this derived information could be added to templatedata. So, the difference from opt-in/opt-out is whether a template author adds this information or whether software adds this information.

How are DOM fragments identified in the output document?
Parsoid uses special markup for transclusions and extensions. T114072 and T105845 propose / request special markup for other DOM fragments. But, looks like this special markup for DOM fragments is a cross-cutting concern. A parser might add additional information (ex: Parsoid in the private data-parsoid attribute) to aid incremental parsing. But, in all cases, it makes sense to come up with output markers that delineate DOM fragments and provide additional information about how they were generated, etc.

Generating DOM fragments: Caveats
For templates that generate a single table row/cell, we need to figure out a way to make sure that parsing to full-dom doesn't break it. i.e. if a template produces  foo  , you cannot parse this string via a HTML5 parser without adding surrounding So, within HTML5, you cannot nest a list within a paragraph. You cannot insert content in a table anywhere except within ,  , and tags. Content inserted anywhere else within a table gets moved out of the table (the content is adopted by the