Parsoid/About

Wikitext has always been both MediaWiki's edit interface and storage format. It has been a great success: the simplicity of wikitext made it possible to start writing Wikipedia with Netscape 4.7 when WYSIWYG editing was technically impossible. A relatively simple PHP script converted the wikitext to HTML.

About 12 years later, the world has changed a bit. Wikitext makes it very difficult to implement visual editing, which is now supported in browsers for HTML documents. It has also become a speed issue: With a lot of new features in the runtime, the conversion from wikitext to HTML can be very slow. For large Wikipedia pages, it can take up to 40 seconds to render a new version after the edit has been saved.

The Wikimedia Foundation's Parsoid project is working on these issues by complementing existing wikitext with an equivalent HTML5 version of the content. In the short term, this HTML representation lets us use HTML technology for visual editing. In the longer term, using HTML as the storage format can eliminate conversion overhead when rendering pages, and can also enable more efficient updates after an edit that only affect part of the page. This might all sound pretty straightforward. So why has this not been done before?

Lossless conversion between wikitext and HTML is really difficult
For the wikitext and HTML5 representations to be considered equivalent, it should be possible to convert between wikitext and HTML5 representations without introducing any semantic differences. It turns out that the ad-hoc structure of wikitext makes such a loss-less conversion to HTML and back extremely difficult.

(insert: A simple example showing that Wikitext is not context-free)
 * Context-sensitive parsing: Wikitext is not context-free, so it cannot be completely described and parsed based on a context-free grammar. The only complete specification of Wikitext's syntax and semantics is the MediaWiki PHP-based runtime implementation, which is still heavily based on regular expression driven text transformation.
 * Text-based templating: The PHP runtime supports an elaborate text-based preprocessor and template system. This works very similar to a macro processor in C or C++, and creates very similar issues. As an example, there is no guarantee that the expansion of a template will parse to a self-contained DOM structure. In fact, there are many templates that only produce a table start tag, a table row  or a table end tag . They can even only produce the first half of an HTML tag or wikitext element (, which is practically impossible to represent in HTML. Despite all this, content generated by an expanded template (or multiple templates) needs to be clearly identified in the HTML DOM.
 * No invalid wikitext: Every possible wikitext input has to be rendered as valid HTML - it is not possible to reject a user's edit with a "syntax error" message. Wiki constructs and HTML tags can be freely mixed in a tag soup, which still needs to be converted to a DOM tree that ideally resembles the user's intention. The behavior for rare edge cases is often more accident than design. Reproducing the behavior for all edge cases is not feasible nor always desirable. We use automated round-trip testing on 100000 Wikipedia articles[LINK?], unit test cases and statistics on Wikipedia dumps to help us identify the common cases we need to support.
 * Character-based diffs: MediaWiki uses a character-based diff interface to show the changes between the wikitext of two versions of a wiki page. Any character difference introduced by a round-trip from wikitext to HTML and back would show up as a dirty diff, which would annoy editors and make it hard to find the actual changes. This means that the conversion needs to preserve not just the semantics of the content, but also the syntax of unmodified content character-by-character. Put differently, since wikitext-to-HTML is a many-to-one mapping where different snippets of wikitext all result in the same HTML rendering (Example: " * list " versus " *list "), a reverse conversion would effectively normalize wikitext syntax. However, character-based diffs forces the wikitext-to-HTML mapping to be treated as a one-to-one mapping. We use a combination of complementary techniques to achieve clean diffs:
 * we detect changes to the HTML5 DOM structure and use a corresponding substring of the source wikitext when serializing an unmodified DOM part (selective serialization).
 * we record variations from some normalized syntax in hidden round-trip data (Ex: excess spaces, variants of table-cell wikitext).
 * we collect and record information about ill-formed HTML that is auto-corrected while building the DOM tree (Ex: auto-closed inline tags in block context).

How we tackle these challenges with Parsoid


Parsoid is implemented as a node.js-based web service. There are two distinct, and somewhat independent pieces to Parsoid: the parser and runtime that converts wikitext to HTML, and the serializer that converts HTML to wikitext.

Converting wikitext to HTML
The conversion from wikitext to HTML DOM starts with a PEG-based tokenizer, which emits tokens to an asynchronous token stream transformation pipeline. The stages of the pipeline effectively do two things:
 * Asynchronous expansion of template and extension tags: We are using MediaWiki's web API for these expansions, which distributes the execution of a single request across a cluster of machines. The asynchronous nature of Parsoid's token stream transformation pipeline enables it to perform multiple expansions in parallel and stitch them back together in original document order with minimal buffering.
 * Parsing of wikitext constructs on the expanded token stream: Quotes, lists, pre-blocks, p-tags are handled via transformations on the expanded token stream. Each transformation is performed by a handler implementing a state machine. This lets us parse context-sensitive wikitext constructs like quotes. By operating on the fully expanded token stream, we can also mimic the PHP runtime's support for structures partly created by templates, or even multiple templates. An example for this are tables created with a sequence of table start / row / table end templates as in this football article.

Fully processed tokens are passed to a HTML5 tree builder. The resulting DOM is further post-processed before it is stored or delivered to a client. The post-processing identifies template blocks, marks auto-corrected HTML tags, and maps DOM subtrees to the original source wikitext range that generated the subtrees. These techniques enable the HTML-to-wikitext reverse transformation to be performed while minimizing dirty diffs.

Converting HTML to wikitext
The conversion from HTML DOM to wikitext is performed in a serializer, which needs to make make sure that the generated wikitext parses back to the original DOM. For this, it needs a deep understanding of the various syntactical constructs and their constraints.

A full serialization of an HTML DOM to wikitext often results in some normalization. For example, we don't track if single quotes vs. double quotes are used in attributes. The serializer always uses double quotes for attributes, which will lead to a dirty diff if a single quotes were used in the original wikitext.

To avoid this, we have implemented a serialization mode which is more selective about what parts of the DOM it serializes. This selective serializer relies on access to both the original wikitext and the original DOM that was generated from it. It compares the original and new DOM it receives and selectively serializes only the modified parts of the DOM. For unmodified parts of the DOM, it simply emits the original wikitext that generated those subtrees. Given that most edits to a stable page are likely going to small and minor relative to the original wikitext, this selective serialization mode pretty much eliminates dirty diffs.

An additional problem that both serializers need to contend with is the presence of wikitext-like constructs in text content. The serializers need to escape wikitext-like text content (Ex:

The a-tag itself should be obvious given that the wikitext is a wiki-link. However in addition to wiki links, external links, images, ISBN links and others also generate an a-tag. In order to properly convert the a-tag back to the correct wikitext that generated it, Parsoid needs to be able to distinguish between them. Towards this end, Parsoid also marks the a-tag with the mw:WikiLink property (or mw:ExtLink, mw:Image, etc.). This kind of RDFa markup also provides clients (like the Visual Editor) additional semantic information about HTML DOM subtrees.

Example 2: Wiki link with templated content
Let us now change the wikitext slightly where the link target is generated by a template:

Foo

The HTML generated by Parsoid for this is:

First of all, note that, in the browser, this wikitext will render identically to Example 1 -- so semantically, there is no difference between the two wikitext snippets. However, Parsoid adds additional markup to the link target. The span-tag wrapping the target has an about attribute and an RDFa type. Once again, this is to let clients know that the target came from a template and to let Parsoid serialize this back to the original wikitext. Parsoid also maintains private information for roundtripping in the data-parsoid HTML attribute (original template source in this example). The about attribute on the span lets us mark template output expanding to several DOM subtrees as a group.

The future
Our roadmap describes our plans for the next months and beyond. Apart from new features and refinement in support of the Visual Editor project, we plan to assimilate several Parsoid features into the core of MediaWiki. HTML storage in parallel with wikitext is the first major step in this direction. This will enable several optimizations and might eventually lead to HTML becoming the primary storage format in MediaWiki. We are also working on a DOM-based templating solution with better support for visual editing, separation between logic and presentation and the ability to cache fragments for better performance.

Join us!
If you like the technical challenges in Parsoid and would like to get involved, then please join us in irc://chat.freenode.net/mediawiki-parsoid. You could even get paid to work on Parsoid: We are looking for a full-time software engineer and 1-2 contractors. Join the small Parsoid team and make the sum of all knowledge easier and more efficient to edit, render, and reuse!