Parsoid is an application which can translate back and forth, at runtime, between MediaWiki's wikitext syntax and an equivalent HTML/RDFa document model with enhanced support for automated processing and rich editing. It has been under development by a team at the Wikimedia Foundation since 2012. It is currently used extensively by VisualEditor, Flow, Content Translation and other applications.
On Wikimedia wikis, for several applications, Parsoid is currently proxied behind RESTBase, which stores the HTML translated by Parsoid.
See Parsoid/Setup for detailed instructions. If you use the MediaWiki-Vagrant development environment using a virtual machine, you can simply add the role visualeditor to it and it will set up a working Parsoid along with Extension:VisualEditor.
If you need help or have questions, you can contact us in the wikitext-l mailing list. If all that fails, you can also contact us by email at parsing-team at the wikimedia.org domain.or
The Parsoid web API + DOM Spec
See Parsoid/API for the web API
See Parsoid/MediaWiki DOM spec to make sense of the HTML that you get from the API. This is the wiki content model spec using HTML/XML DOM and RDFa and is designed to be useful as a future storage format.
Running the tests
To run all parser tests and mocha tests:
parserTests has quite a few options now which can be listed using
node parserTests --help.
Converting simple wikitext
You can convert simple wikitext snippets from the command line using the parse.js script in the tests/ directory:
echo '[[Foo]]' | node parse
The parse script has a lot of options.
node parse --help gives you information about this.
In Ubuntu 13 and 14,
node has been renamed to
nodejs. There, either create a symbolic link (or equivalent) or type:
echo '[[Foo]]' | nodejs parse
Debugging Parsoid (for developers)
See Parsoid/Debugging for debugging tips.
- Internals: documentation about Parsoid internals with links to other details.
- Parsoid deployment agenda on Wikimedia cluster (code normally deployed every Monday and Wednesday between 1pm - 1:30pm PST)
- Parsoid/Round-trip testing: The round-trip testing setup we are using to test the wikitext -> HTML DOM -> wikitext round-trip on actual Wikipedia content.
- Parsoid/Visual Diffs Testing: Info about visual diff testing for comparing Parsoid's html rendering with php parser's html rendering + a testreduce setup for doing mass visual diff tests.
- Parsoid/limitations: Limitations in Parsoid, mainly contrived templating (ab)uses that don't matter in practice. Could be extended to be similar to the preprocessor upgrade notes (Might need updating)
- Parsoid/Roadmap: old Parsoid roadmap. (NOW STALE and needs updating)
- Parsoid/Bibliography: Bibliography of related literature
- RT testing commits (useful to check regressions and fixes)
- Deployment instructions for Parsoid
- Kibana dashboards: all events, CPU timeouts, request timeouts
- Grafana dashboard for wt2html metrics
- Grafana dashboard for html2wt metrics
- Grafana dashboard for non-200 responses
- Ganglia dashboard for Parsoid cluster
- See Parsoid/Debugging for debugging tips.
- Upgrading or adding packages to Parsoid
- Jenkins Job Builder docs for updating jenkins jobs