Parsing/Notes/Wikitext 2.0

Wikitext 2.0: Typed wikitext (?)
Think of this as retroactively imposing a type system on top of the existing wikitext markup language. Editors don't need to know about any of this typing except when they want to control how transclusions behave / compose with the surrounding context. This is purely a layered abstraction on top of the string-based model and as such won't involve any drastic changes to wikitext itself but which could lay the path for changes in the future.

Typed wikitext constructs

 * Wikitext constructs (be it a piece of text, links, list markup, nowiki markup, extensions, transclusions, etc.) returns a string, but are interpreted as a typed value. The available types are:
 * 1) string
 * 2) a key value pair
 * 3) a dom tree
 * More generally, 2. and 3. would be a map and a dom-forest. We might also expand 3. into a bunch of specialized types for reasons that will become clear later on. But in the interest of simplicity, let us ignore these details now.


 * The types are treated as constraints that are enforced on the output of the wikitext construct, i.e. if a template declares its output to be a dom tree, then the output of the template will  get that type enforced. However, if the output of the template  is declared to be a string, all html tags inside it will be  appropriately escaped to render as a string. So, for example, for use cases where HTML attributes need to  be templated, you can only use constructs that return a string  or a key-value pair. If a DOM-tree typed construct is used, it  could be (silently or loudly with error markup in output) ignored.

Document composition spec / rules

 * The result of this typed wikitext notion is that for all dom-tree returning constructs, we now have a bunch of dom fragments that need to be composed together into a top-level document. This composition  is trivial in the non-nesting cases. However, where there is nesting,  we also need to specify fragment composition rules to specify how  composition is going to work (see document composibility). One way of resolving this is to not specify any composition rules  in the wikitext spec and let HTML tree building resolve nesting  scenarios according to the HTML5 content model. This is the  wikitext 1.0 status quo but we can do much better than that and  build upon it. The balanced templates RFC is proposing more finer grained types for output of templates that de facto defines how the template  output is going to compose with the surrounding context.


 * Markup errors are going to continue to be an issue. The DOM tree type for a construct implicitly enforces fixups of unclosed and misnested tags within the output of the construct (ex: an unclosed or  other inline/phrase tags are going to get implicitly fixed up within  list items, paragraphs, table cells, etc. and won't leak out of those  contexts). However, we would nevertheless benefit from notion of top-level  dom-scopes which bounds the range of unclosed tags. One way of doing this is defining an implicit top-level section construct (much like the implicit paragraph wikitext construct) whose output is naturally a DOM tree. This takes care of unclosed tags within a section. But, this potentially eliminates one use case that seems to be used on wikis which is wrapping the entire page or wrapping multiple sections in  tags. We need to figure out what that fits into this model.

Metadata as annotations on the DOM tree

 * In addition to returning a typed value, a construct can also return metadata about the construct / document / page. Ex: categories, refs, lang links, etc. The metadata is represented as annotations  on nodes of a DOM tree. This is a detail that we can ignore for  the moment - to be elaborate upon later.

Implications of this model
There are a bunch of implications that naturally fallout from the above model / layer.

Decoupling top-level document from extensions and templates

 * The top-level document becomes independent of extensions and templates, i.e. the top-level markup can be seen as producing a document with holes into which output of extensions and templates will get slotted in. So,    will *always* parse to a bold tag no matter what   outputs. Since the template output is nested within the  tag, document composition rules for nesting will determine how the composed document looks like, but it is not going to alter the fact that at the top level, both foo and bar will be bolded.
 * This is a big change from wikitext 1.0 where you cannot parse the document till all transclusions are expanded. This has implications for performance as well (you can parse all the top-level document, templates, and extensions in parallel and compose the output together by relying on document composition rules). This has implications for wikitext reasoning for editors as explained earlier.


 * You can cache the output of templates and extensions separate from the page they were found on (with some caveats where they aren't relying on things like time of day, or other such volatile components). For example, can be cached and reused wherever it occurs.


 * Transclusions, extensions, gadgets, are all identical. They all fill DOM-shaped holes in the top-level document. For all practical purposes, a transclusion can be modelled as an extension that has its own special-purpose shortcut syntax  instead of.

Clearer spec for nesting rules

 * The ability to defining and spec composition rules let us deal with pesky problems like links-nested-in-links (common on wikis) and fosterable-content-in-tables(also common on wikis) which introduce a fair bit of complexity in Parsoid and a lot of known rendering errors on wikis.
 * For example, we could spec a behavior where we specify that when content is nested inside a link, all links within the nested content will be stripped.
 * Similarly, we could spec behavior where we specify that when content is nested inside but outside a table-cell, (a) we will implicitly add any necessary table rows / cells to hold that content (b) we will quietly swallow the content and not render it (c) we will swallow the content but render error markup so editors can fix it.

Well-structured / specified output (?)

 * The DOM tree typing make supporting tools like VE and CX simpler without the kind of contortions that Parsoid currently goes through.
 * We can continue to use (a variation) of the Parsoid DOM spec that continues to provide semantic markup on special constructs.
 * With some carefully crafted composition rules, you can even view the original wikitext as a nested tree which benefits source editing tools.

Defining an extension spec

 * Since extensions are going to be useful and will continue to be developed, we need to define an extension spec that is not dependent on the internals of the parsing tools. In the common case, all extensions need to do is the normal thing, i.e. they take some input and produce a typed value (a DOM tree in the common case) which gets composed into the document.
 * But, there will continue to be extensions like cite that need to do some post-processing on the document. So, they can be provided hooks wherein they get to post-process the document. For example, the extension could process metadata annotated to the dom tree to collect &#x3C;ref>s and generate the appropriate &#x3C;ref> links and generate &#x3C;references> output (this is how Parsoid's Cite implementation is structured). So, the spec would also need to provide mechanisms to attach metadata to the source document.
 * There may be other use cases that fall out by examining the existing extensions and how they are used.
 * Extensions like &#x3C;translate&#x3E; could potentially be modelled as tools that decorate the DOM tree with metadata annotations. These metadata annotations can then be used by editing tools to provide translation interfaces.

Separating raw parser output from final mediawiki output

 * It would be useful to separate the raw parser output from the final mediawiki output, i.e. the parser does't concern itself with database state (red links, bad images, resource sizes, etc.). It produces output that is then post-processed to generate the final mediawiki output. This further decouples the parser from mediawiki internals and lets different platforms (mobile, desktop, etc.) define their own post-processing transformations. This might be an architectural / processing model distinction primarily and the implementation might be structured differently.

Getting there

 * Since the value types as well as the document composition rules are layered on top of base wikitext 1.0, we can rely on the content handler abstraction to give us a fairly clean way of transitioning to the newer model.
 * If a page declares itself to be a wikitext 2.0 page, then this abstraction layer kicks in. In practice, the page will be handled by a totally different code path.
 * But, this layering also lets us use today's templates on both wikitext 1.0 and wikitext 2.0 changes and gradually migrate templates and pages over. This is based on the observation that in practice, a larger proportion of templates are wikitext 2.0 compliant, i.e. they return well-formed DOMs and in many cases, document composition is trivial without requiring any fixups.
 * In addition, Parsoid has sufficient information about existing templates and their uses to identify where we would need to either develop new templates or tweak existing templates to make them usable / compliant with wikitext 2.0
 * We can rely on 'undefined behavior' spec magic to let grey areas slide while also providing editors a good way to fix such behavior.