Parsing/Notes/Wikitext 2.0/Strawman Spec

Here is a somewhat tentative strawman spec for how we might implement the ideas in Parsing/Notes/Wikitext 2.0 and are based on prior ideas in User:SSastry (WMF)/Notes/Wikitext.

Note to discerning readers: This is not as much a spec as it is an outline of an implementation. But, a spec can be extracted from this. But, yes, the "Spec" part of the title is probably misleading.

Goals

 * Improve ability to reason about wikitext
 * Reduce edge cases by bounding range of wikitext errors
 * Improve editability in VE
 * Improve performance

Parsing outline / spec

 * Remove the notion of preprocessing from the core parser.
 * Top-level page is structurally parsed for wikitext-native constructs as follows:
 * Parsing the document yields an AST with 2 classes of nodes.
 * those that are parsed to an independent DOM fragment.
 * those that are not parsed to an independent DOM fragment on their own.
 * Paragraphs, indent-pres, links, quotes are not processed to DOM. The wikitext markup is replaced with the equivalent HTML markup.
 * Possible enhancement: Consider link content to be a nested document and balance it independently (Parsoid does this right now already).
 * Headings, top-level sections tables, list items, transclusions, extension tags, are parsed to DOM (recursively bottom-up) and the AST node is replaced with a DOM-fragment.
 * Special cases:
 * For table nodes, fostered content is easily identifiable without heroic analyses and can be handled appropriately according to whatever semantics we want.
 * error markup to get editors to fixup.
 * silently discard fostered content.
 * silently display fostered content but prevent direct visual editing and flag the error in VE.
 * error markup during preview.
 * ... other possibilities? ...
 * Templates can specify one of multiple types for their output. These types are enforced in the composition step below.
 * Plain text
 * Attribute pairs
 * DOM forest
 * more specialized DOM types (ex: inline, block, table, etc. as in T114445)
 * This transforms the AST into an AST tree where every node is a DOM fragment.
 * Rely on composition rules to transform these DOM fragments into a DOM tree. This composition spec needs to be developed.
 * Simplest "do nothing" rule (which is wikitext 1.0 by default): All the dom fragments are flattened to a html string and the HTML is reparsed into the final DOM. This relies on HTML5 tree builder algorithm to fix things up.
 * Some possible composition rules:
 * All content nested within links will have links stripped
 * If a p-tag is nested in another p-tag, transform DOM in a well-defined manner (easy to work out the detail, skipping here for now).
 * Others based on
 * ... more to be worked out based on perusing the HTML5 tree building spec and common-sense semantic notions of nesting ...
 * Run Add template wrapping markup + other DOM passes (will be a subset of Parsoid's DOM as necessary).
 * Sanitize (last step before serialization)
 * XML serialize.

Technical Blockers

 * Use of parser functions at the top-level (vs. in templates / extensions)
 * not sure sure how to tackle this yet
 * Templates that heavily rely on preprocesing:
 * Preprocessing-based templates can be processed with a specialized extension that yields fully processed markup which is then parsed using the above algorithm.
 * In the long run, templates might be processed recursively using the same algorithm, but for now, for templates that rely on preprocessing, this is not feasible.
 * So, this is only a blocker in that the extension needs to be written.
 * Multi-template authoring of DOM structures (tables, 2-column layout, etc.) where each individual template wouldn't generate a well-formed DOM structure.
 * One way of handling it is using heredoc style syntax that enables these to be written using an atomic syntactical construct. Ex: T114432
 * Deprecate and remove all parser hooks that are parser-internals specific
 * Doesn't look too bad according to Parser_Hooks_Stats
 * Come up with an extension spec that extension authors can do to implement the functionality currently used by various extensions
 * This is a matter of working out the details
 * Actually reviewing the sanity of this proposal, working out the kinks, prototyping this, and evaluating this.

Social Blockers
I am just doing a first cut laundry list here. These are all genuine and valid concerns. We need to think through these and understand real impacts. My gut sense is that the changes are not going to be drastic. A lot of wikitext will continue to behave as before, but there might be some subtle changes (for the good, I think). But, template authors are probably the ones that might have the biggest concerns here.
 * Potential concerns about breakage about pages
 * Potential concerns about templates that need updating
 * Potential concerns about changes to editing workflow
 * Potential concerns about changes to well-understood wikitext semantics.
 * How does this affect bots?

Advantages
Parsing/Notes/Wikitext 2.0 elaborates on some possible benefits. Repeating them here in a concise form for reasons of completeness of this document.
 * Improved reasoning abilities: clear semantics and improved reasoning about wikitext markup.
 * Top-level document is decoupled (in good ways) from transclusions and extensions. So,  will always parse as an italic even if there were transclusions in some wikitext here.
 * Uniform semantics for templates, extensions, gadgets, whatever.
 * Bounding effects of markup errors:
 * markup errors are bound to and fixed within the smallest possible scopes.
 * Potential for improved edit tooling:
 * Structural semantics of the input and output document is an enabler for improved editing tools.
 * You can enable micro-editing at the sub-section level without edit conflicts as long as the nodes being edited are different. You can imagine locking / warning / timeout / real time editing features built on top of this.
 * Potential for improved parsing performance:
 * Templates can be parsed independent of the top-level document and concurrently.
 * As long as templates don't use features that depend on volatile output (time of day, etc.), their output can be cached.
 * Both of these have tremendous performance implications for template edits. If you fix a typo in a heavily used template, you parse the template output once, cache it, and (with some caveats), in most pages that use the template, you can do a drop-in replacement of the html string in place of the old string and be done.
 * Other incremental parsing abilities on edits. If a typo is fixed on a page, there is absolutely no reason to reparse the entire page. It is conceivable to come up with solutions / algorithms to fix the typo in the HTML output.