Parsoid/Roadmap

After the successful December release and some more clean-up work following up on that we are now considering the next steps for Parsoid. The medium-term plan for the summer is to support VisualEditor (VE) in becoming the default editor on all wikipedias.

The main tasks we see on the Parsoid side to get there until July are:


 * Performance improvements: Loading a large wiki page through Parsoid into VisualEditor can currently take over 30 seconds. We want to make this instantaneous by generating and storing the HTML after each edit. This requires a throughput that can keep up with the edit rates on major wikipedias (~10 Hz on enwiki).


 * Features and refinement: Localization support will enable the use of Parsoid on non-English wikipedias. VisualEditor needs editing support for more content elements including template parameters and extension tags. As usual, we will also continue to refine Parsoid's compatibility in round-trip testing and parserTests.

Apart from these main tasks closely connected to supporting VE, we also need to look at the longer-term Parsoid and MediaWiki strategy. Better support for visual editing and smarter caching in MediaWiki's templating facilities is one area we plan to look at. We also would like to make it easy to use the VisualEditor on small mediawiki installations by removing the need to run a separate Parsoid service.

The general idea is to push some of Parsoid's innovations back into MediaWiki core. The clean and information-rich HTML-based content model in particular opens up several attractive options. Re-parsing a page after an edit can be sped up by reusing the parser output for unmodified parts of the page. We plan to implement this in Q2 2013, and expect a significantly higher parser throughput with this feature. Similarly, updates triggered by template modifications or parser function changes (changes in date or time, for example) can be made more efficient by only re-expanding affected parts of the page (Q3).

At the same time, we have decided to narrow our focus a bit by continuing to use the PHP preprocessor to perform our template expansion. This gives us complete coverage of preprocessor functionality including Lua integration, and lets us expand templates in parallel by doing concurrent API calls. Parsoid's template expansion pipeline works very well too, but doesn't implement all parser functions natively and lacks Lua integration.

For more discussion of the technical decisions we have made, see Parsoid/reasoning behind Q1 2013 technical decisions.

Features: Editing support for images and categories, localization
Editing support for images needs to be implemented, categories are already supported but need testing.

Localization support for non-English wikis (namespaces, magic words, link trails and -prefixes, language variants etc) will be developed. Per-wiki configuration information is retrieved through the API (implemented). The HTML DOM interface abstracts the localization issues for the VisualEditor.

Performance: Generate and store HTML DOM on edit
Instead of converting a (potentially large) wiki page to HTML when a user loads the page into the VisualEditor, we will do so in the background after each edit. The result will be stored in the database, which will make loading a page into the VisualEditor practically instantaneous since no more conversion needs to be performed.

The HTML/RDFa DOM content model we developed is aiming to be an equivalent representation of the content. It can contain fully expanded templates while still providing the metadata needed to re-expand a template later. This makes the HTML DOM an equivalent representation of a revision with the added capability to persistently cache template expansions and extension output inline. The inline cache enables further performance improvements for subsequent edits and refreshLink jobs, which we describe further down in this document.

Adding HTML storage will probably involve adding an additional text table and adapting the regular Revision storage logic to optionally use this. Storage space itself does not seem to be an issue (todo: double-check with ops!). The same text id as the corresponding wikitext can be used in the HTML table to avoid any schema changes in the revision table.

Performance: Batch API calls
We currently naïvely perform all API requests in parallel. With most API calls returning very quickly, this wastes resources on Parsoid as well as API servers. Instead, we will batch several requests into a single larger one to amortize the setup and connection costs. See bug 43888 for the details.

Testing / good to have: Start recording performance data from round-trip testing
For capacity planning and optimization progress tracking we need performance information on as many pages as possible. It should not be too hard to extend our round-trip testing infrastructure to collect this information. We will probably not have time for this project ourselves, but it is quite self-contained and well-suited as a project for an external contributor.

Features: Editing support for citations, template parameters and tag extensions
The main focus is on making citations and their associated templates editable, so that VisualEditor users can properly reference their sources. We will rework our version of the Cite extension to support dynamic re-expansion of the references tag. This will be needed both on the server side (for incremental updates) and the client side (inside the VisualEditor, potentially).

Template parameter editing and extension tag editing will be wikitext-based. This accommodates unbalanced template parameters, which are sadly relatively common in existing content. Both parameters and extension tag bodies will restricted syntactically, so that wikitext edits in these cannot affect other parts of the page.

Features: HTTP API to render extension tags directly
We currently use an action=parse API hack to expand extension tags to HTML. Instead of this hack, we want to add a dedicated extension tag expansion end point that can also be used by the VisualEditor to update / insert extension tags inline.

Performance: Incremental re-parsing after wikitext edit
After an edit to a wiki page using the wikitext UI, we currently re-parse the entire page. In most cases only a small part of the page was actually modified, so a full re-parse is not really needed.

Using the DSR (DOM source range) information stored in the HTML DOM, we can match the position of a wikitext diff to a containing HTML DOM structure and re-parse only the modified version of that node. This would normally be a top-level element like a paragraph, which does not depend on nested parser state for correct rendering. Expensive operations like template expansions would normally not need to be re-performed, which would make parse times proportional to the edit size rather than the page size.

There are some cases where a wikitext diff inside a block element can affect other top-level blocks when tokenized as a full page. We need to rule this case out before applying this optimization. Similar heuristics as we use in wikitext escaping should work in this case too.

Research / prototype: HTML-only wiki support
The Parsoid web service adds a complex dependency to MediaWiki installations, which is problematic for simple MediaWiki installations that just want to use the VisualEditor. Wikis interested in editing through the VisualEditor exclusively don't necessarily need wikitext-based storage. Instead, they could use HTML storage natively. We already intend to add the capability for HTML storage in MediaWiki, which makes the storage part relatively easy.

In addition, HTML-only wikis will need a HTML-based diff implementation similar to the one in localwiki or the XyDiff XML diff algorithm to replace the wikitext source-based diff.

We will investigate which other issues we need to solve to make an HTML-based wiki possible.

Research / prototype: DOM-based templating
HTML-only wikis might want to provide similar templating functionality as the existing wikitext-based template system. This could be DOM-based.

The main things we need are
 * Expressions: provide access to modules and logic, but cannot define infinite loops or variables
 * Iteration: Iterate over finite data structures (JSON objects for example)
 * Conditionals: Include / evaluate a sub-DOM depending on an expression
 * Variable interpolation in attributes and text content

This functionality is pretty simple to implement on the DOM (possibly using JS/JQuery or XPath). It would provide an opportunity to define very minimal service-like (RESTful for example) extension interfaces, which extensions could port to for a gradual transition.

One popular option is to embed control structures in attributes similar to TAL, Distal or Genshi. Another option is to provide a separate binding to a plain HTML document as in Pure. Templates themselves would still be valid HTML, which might make it possible to implement some sort of visual editing mode for templates. Any serious logic would live in Lua or JavaScript modules working on DOM fragments, JSON objects or strings instead of being embedded in the template itself. The limited expressions supported by ESI might serve as an inspiration.

Features: Support HTML-only wikis without Parsoid
We will support simple HTML-only wikis with VisualEditor front-end without the need for a Parsoid installation.

Performance: Fragment caching and incremental updates
Parsoid encapsulates parts of the DOM generated from template expansions, extensions etc. It can be used to classify current templates in those emitting self-contained DOM output (properly nested) and those emitting just a start or end tag (table start / row / end templates for example). Fortunately, most templates produce properly nested output. Those that don't can be marked with a flag in the database, after which proper nesting can be enforced for all other templates from there on. Unbalanced templates are encapsulated in a combined DOM block, which is then properly nested again. This can also be enforced when re-expanding the combined block of templates.

With proper nesting enforced and all template parameters available re-rendering a template will only swap out a DOM subtree. This makes it possible to cache fast-changing templates or extension output (WikiData infoboxes for example) as a fragment in the edge caches, the DB or update it dynamically in clients.

With more per-fragment metadata (reference counted links and list of recursively used templates), the LinkUpdate jobs can be restricted to a re-expansion of the affected template transclusions rather than the full page. The general idea is to collect all dependencies during evaluation, and encode this information efficiently (likely outside the DOM) to enable quick dependency and validity checks.

Some extensions like Cite use global state, for example to number citations. Sadly, this complicates independent re-expansions. It does however seem to be possible to implement numbering and similar page-wide operations using CSS and/or JS, which would also benefit the VisualEditor. Most other extensions used by the WMF like math, poem, timeline etc are order-independent, so this seems to be a solvable issue for the extensions we currently care about.

Parsoid can (and does in the VE deployment) use the PHP preprocessor via the 'expandtemplates' web API method. This lets it fully support parser functions, internal PHP interfaces, Lua scripting etc without having to re-implement this functionality. The result is pre-expanded wikitext, which is then parsed and encapsulated in Parsoid. Tag extensions are expanded independently (via an action=parse API call currently).

In the longer term, we could extend the PHP API to provide more dependency information along with the expanded output. A list of templates, parser functions and Lua scripts used in the expansion would provide pretty complete dependency information for caching / incremental update purposes.

Treating the PHP preprocessor and its associated extensions as a self-contained 'legacy' component side-steps the problems associated with the wikitext-centric interfaces used. Emulating wikitext-based parameters and frame objects passed to (for example) Lua from a token-based parser will probably never work perfectly and involve a lot of work. Performance of template expansions should not matter that much with incremental updates, as they would be relatively rare. For new pages, all template expansions can be performed in parallel (Parsoid currently sends one parallel API request per transclusion), which could be refined with some batching to amortize fixed connection overheads.

If still necessary for performance: Fast and integrated C++ implementation
If all of the above fails to produce the required performance (unlikely by our calculations), then we can still dust off the started C++ port and move some part or all of Parsoid to C++.