Parsoid

From MediaWiki.org
Jump to navigation Jump to search
Artist's impression of the Parsoid HTML5 + RDFa wiki runtime

Parsoid is an application that allows for converting back and forth between wikitext and HTML. The original application is written in JavaScript (using Node.js) and was operational on the Wikimedia cluster between Dec. 2012 and Dec. 2019. In 2019, Parsoid was ported to PHP, and the PHP version replaced the JS version on the Wikimedia cluster in Dec. 2019. Parsoid is meant to be eventually integrated into core MediaWiki, with the goal of replacing MediaWiki's current native parser.

Parsoid (the PHP version) is planned to come natively bundled with MediaWiki by June 2020. At that time, the setup instructions on this page will be updated. Until then, for non-Wikimedia installations, Parsoid/JS is the supported version of Parsoid.

Technical details[edit source]

Parsoid is an application which can translate back and forth, at runtime, between MediaWiki's wikitext syntax and an equivalent HTML/RDFa document model with enhanced support for automated processing and rich editing.

It has been under development by a team at the Wikimedia Foundation since 2012. It is currently used extensively by VisualEditor, Flow, Content Translation and other applications.

Parsoid is intended to provide flawless back-and-forth conversion, i.e. to avoid information loss and also prevent "dirty diffs".

On Wikimedia wikis, for several applications, Parsoid is currently proxied behind RESTBase, which stores the HTML translated by Parsoid.

For more on the overall project, see this blog post from March 2013. To read about the HTML model being used, see MediaWiki DOM spec.

Parsoid was originally structured as a web service and written in JavaScript, making use of Node.js. It is currently being ported to PHP; for more information see this tech talk from February 2019 (slides).

GitHub Repository: https://github.com/wikimedia/parsoid

Usage[edit source]

Development[edit source]

Development happens in the Parsoid service Git repository. Code review happens in Gerrit. See Gerrit/Getting started to set up an account for yourself.

If you use the MediaWiki-Vagrant development environment using a virtual machine, you can simply add the role visualeditor to it and it will set up a working Parsoid along with Extension:VisualEditor.

Note that the most-recently released version of Parsoid is written in JavaScript, and installation of Parsoid/JS is what is described below. This is what you should use if you are running MediaWiki 1.34 or earlier. Current development of Parsoid is happening on the version written in PHP, called Parsoid/PHP. See the Developer Setup instructions if you would like to install that version of Parsoid, which requires the latest development version of MediaWiki.

Running the tests[edit source]

To run all parser tests and mocha tests:

$ npm test

The parser tests have quite a few options now which can be listed using node bin/parserTests.js --help. To run tests on the PHP port:

$ composer test

Converting simple wikitext[edit source]

You can convert simple wikitext snippets from the command line using the parse.js script in the bin/ directory:

echo 'Foo' | node bin/parse.js

The parse script has a lot of options. node bin/parse.js --help gives you information about this.

(In Ubuntu 13 and 14, node has been renamed to nodejs. There, either type nodejs or create a symbolic link or equivalent.)

Debugging Parsoid (for developers)[edit source]

See Parsoid/Debugging for debugging tips.

Technical documents[edit source]

Links for Parsoid developers[edit source]

Links for Parsoid deployers (to the Wikimedia cluster)[edit source]

See also[edit source]

Contact[edit source]

If you need help or have questions/feedback, you can contact us in #mediawiki-parsoid connect or the wikitext-l mailing list. If all that fails, you can also contact us by email at parsing-team at the wikimedia.org domain.