Parsoid



Parsoid is a library that allows for converting back and forth between wikitext and HTML. The original application is written in JavaScript (using Node.js) and was operational on the Wikimedia cluster between Dec. 2012 and Dec. 2019. In 2019, Parsoid was ported to PHP, and the PHP version replaced the JS version on the Wikimedia cluster in Dec. 2019. Parsoid is meant to be integrated into core MediaWiki, with the goal of eventually replacing MediaWiki's current native parser.

Parsoid (the PHP version) is natively bundled in MediaWiki 1.35, released in August 2020. For non-Wikimedia installations, Parsoid/JS will be supported until the end-of-life of MediaWiki 1.31 (LTS) in June 2021.

Technical details
Parsoid is an application which can translate back and forth between MediaWiki's wikitext syntax and an equivalent HTML/RDFa document model with enhanced support for automated processing and rich editing.

It has been under development by a team at the Wikimedia Foundation since 2012. It is currently used extensively by VisualEditor, Flow, Content Translation and other applications.

Parsoid is intended to provide flawless back-and-forth conversion, i.e. to avoid information loss and also prevent "dirty diffs".

On Wikimedia wikis, for several applications, Parsoid is currently proxied behind RESTBase, which stores the HTML translated by Parsoid. It is expected that RESTBase will eventually be replaced with a cache more tightly integrated with MediaWiki.

For more on the overall project, see this blog post from March 2013. To read about the HTML model being used, see MediaWiki DOM spec.

Parsoid was originally structured as a web service and written in JavaScript, making use of Node.js. A tech talk from February 2019 and blog post describes the porting process. The Parsoid extension API is currently under active development; a tech talk from August 2020 describes this work.

GitHub Repository: https://github.com/wikimedia/parsoid

Usage

 * /Releases/ - List of releases made for parsoid
 * /Setup
 * /Troubleshooting
 * /API - for the web API
 * MediaWiki DOM spec - to make sense of the HTML that you get from the API, designed to be useful as a future storage format
 * /LanguageConverter - notes on Parsoid's implementation of LanguageConverter
 * /Known differences with PHP parser output

Development
Development happens in the. Code review happens in Gerrit. See Gerrit/Getting started to set up an account for yourself.

If you use the MediaWiki-Vagrant development environment using a virtual machine, you can simply add the role  to it and it will set up a working Parsoid along with Extension:VisualEditor. (This may have been broken by the switch to Parsoid/PHP: T258940)

Note that the most-recently released version of Parsoid is written in PHP, and installation of Parsoid/PHP is what is described below. This is what you should use if you are running MediaWiki 1.35 or later. Check Parsoid/JS if you are running the old version of Parsoid written in JavaScript, and used for MW 1.34 and earlier.

Running the tests
To run all parser tests and mocha tests: The parser tests have quite a few options now which can be listed using.

If you have the environment variable  pointing to a configured MediaWiki installation, you can run some additional tests with:

Converting simple wikitext
You can convert simple wikitext snippets from the command line using the  script in the   directory:

echo 'Foo' | php bin/parse.php

The parse script has a lot of options. gives you information about this.

Debugging Parsoid (for developers)
See Parsoid/Debugging for debugging tips.

Technical documents

 * /Internals: documentation about Parsoid internals with links to other details.
 * PHP Porting notes and help-wanted tasks
 * Parsoid deployment agenda on Wikimedia cluster (code normally deployed every Monday and Wednesday between 1pm - 1:30pm PST)
 * /Round-trip testing: The round-trip testing setup we are using to test the wikitext -> HTML DOM -> wikitext round-trip on actual Wikipedia content.
 * /Visual Diffs Testing: Info about visual diff testing for comparing Parsoid's html rendering with php parser's html rendering + a testreduce setup for doing mass visual diff tests.
 * /limitations: Limitations in Parsoid, mainly contrived templating (ab)uses that don't matter in practice. Could be extended to be similar to the preprocessor upgrade notes (Might need updating)
 * /Bibliography: Bibliography of related literature

Links for Parsoid developers

 * See Parsoid/Debugging for debugging tips.
 * Upgrading or adding packages to Parsoid
 * See these instructions for syncing Parsoid's copy of parser tests to/from core
 * Parsoid has a limited library interface for invoking it programatically.

Links for Parsoid deployers (to the Wikimedia cluster)

 * Parsoid/Deployments
 * RT testing commits (useful to check regressions and fixes)
 * Deployment instructions for Parsoid
 * Kibana dashboard
 * Grafana dashboard for wt2html metrics
 * Grafana dashboard for html2wt metrics
 * Grafana dashboard for non-200 responses
 * Prometheus breakdown for the Parsoid cluster on eqiad
 * Prometheus breakdown for the Parsoid cluster on codfw
 * Jenkins Job Builder docs for updating jenkins jobs

Contact
If you need help or have questions/feedback, you can contact us in or the wikitext-l mailing list. If all that fails, you can also contact us by email at parsing-team at the wikimedia.org domain.