Parsoid
Parsoid
A bidirectional runtime wikitext parser. Converts back and forth between wikitext and HTML/XML DOM with RDFa.
|
Parsoid allows an editor to use MediaWiki extensions such as VisualEditor.
Contents
Technical details[edit]
Parsoid is an application which can translate back and forth, at runtime, between MediaWiki's wikitext syntax and an equivalent HTML/RDFa document model with enhanced support for automated processing and rich editing.
It has been under development by a team at the Wikimedia Foundation since 2012. It is currently used extensively by VisualEditor, Flow, Content Translation and other applications.
Parsoid is structured as a web service and is written in JavaScript, making use of Node.js. It is intended to provide flawless back-and-forth conversion, i.e. to avoid information loss and also prevent "dirty diffs".
On Wikimedia wikis, for several applications, Parsoid is currently proxied behind RESTBase, which stores the HTML translated by Parsoid.
For more on the overall project, see this blog post from March 2013. To read about the HTML model being used, see MediaWiki DOM spec.
GitHub Repository: https://github.com/wikimedia/parsoid
Usage[edit]
- Releases - List of releases made for parsoid
- /Setup
- /Troubleshooting
- /API - for the web API
- MediaWiki DOM spec - to make sense of the HTML that you get from the API, designed to be useful as a future storage format
- /LanguageConverter - notes on Parsoid's implementation of mw:LanguageConverter
- /Known differences with PHP parser output
Development[edit]
Development happens in the Parsoid service Git repository. Code review happens in Gerrit. See Gerrit/Getting started to set up an account for yourself.
If you use the MediaWiki-Vagrant development environment using a virtual machine, you can simply add the role visualeditor
to it and it will set up a working Parsoid along with Extension:VisualEditor.
Running the tests[edit]
To run all parser tests and mocha tests:
npm test
parserTests has quite a few options now which can be listed using node parserTests --help
.
Converting simple wikitext[edit]
You can convert simple wikitext snippets from the command line using the parse.js
script in the bin/ directory:
echo 'Foo' | node parse
The parse script has a lot of options. node parse --help
gives you information about this.
In Ubuntu 13 and 14, node
has been renamed to nodejs
. There, either type nodejs
or create a symbolic link (or equivalent).
Debugging Parsoid (for developers)[edit]
See Parsoid/Debugging for debugging tips.
Technical documents[edit]
- /Internals: documentation about Parsoid internals with links to other details.
- Parsoid deployment agenda on Wikimedia cluster (code normally deployed every Monday and Wednesday between 1pm - 1:30pm PST)
- /Round-trip testing: The round-trip testing setup we are using to test the wikitext -> HTML DOM -> wikitext round-trip on actual Wikipedia content.
- /Visual Diffs Testing: Info about visual diff testing for comparing Parsoid's html rendering with php parser's html rendering + a testreduce setup for doing mass visual diff tests.
- /limitations: Limitations in Parsoid, mainly contrived templating (ab)uses that don't matter in practice. Could be extended to be similar to the preprocessor upgrade notes (Might need updating)
- /Bibliography: Bibliography of related literature
Links for Parsoid developers[edit]
- See Parsoid/Debugging for debugging tips.
- Upgrading or adding packages to Parsoid
- See these instructions for syncing Parsoid's copy of parser tests to/from core
- Parsoid has a limited library interface for invoking it programatically.
Links for Parsoid deployers (to the Wikimedia cluster)[edit]
- Parsoid/Deployments
- RT testing commits (useful to check regressions and fixes)
- Deployment instructions for Parsoid
- Kibana dashboard
- Grafana dashboard for wt2html metrics
- Grafana dashboard for html2wt metrics
- Grafana dashboard for non-200 responses
- Prometheus breakdown for the Parsoid cluster on eqiad
- Prometheus breakdown for the Parsoid cluster on codfw
- Jenkins Job Builder docs for updating jenkins jobs
See also[edit]
- RESTBase: a caching / storing API proxy for page HTML translated by Parsoid
- Quarterly review meetings of the Parsoid team: April 2015, January 2015 (earlier)
- Future/Parser plan: Early (now relatively old) design ideas and issues
- Special:PrefixIndex/Parsoid/: Parsoid-related pages on this wiki
- parsoid-jsapi: a high-level interface for extraction and transformation of wikitext, similar to the mwparserfromhell API.
- Alternative parsers
Contact[edit]
If you need help or have questions/feedback, you can contact us in #mediawiki-parsoid
connect or the wikitext-l mailing list. If all that fails, you can also contact us by email at parsing-team at the wikimedia.org domain.
![]() | Parsoid is maintained by the Readers Team (Parsing Team).
Get help:
|
- WMF Projects
- WMF Projects 2011q4
- WMF Projects 2012q1
- WMF Projects 2012q2
- WMF Projects 2012q3
- WMF Projects 2012q4
- WMF Projects 2013q1
- WMF Projects 2013q2
- WMF Projects 2013q3
- WMF Projects 2013q4
- WMF Projects 2014q1
- WMF Projects 2014q2
- WMF Projects 2014q3
- WMF Projects 2014q4
- WMF Projects 2015q1
- WMF Projects 2015q2
- WMF Projects 2015q3
- WMF Projects 2015q4
- WMF Projects 2016q1
- WMF Projects 2016q2
- WMF Projects 2016q3
- WMF Projects 2016q4
- WMF Projects 2017q1
- WMF Projects 2017q2
- WMF Projects 2017q3
- WMF Projects 2017q4
- WMF Projects 2018q1
- WMF Projects 2018q2
- WMF Projects 2018q3
- WMF Projects 2018q4
- WMF Projects 2019q1
- MediaWiki development
- Parser
- Parsoid
- Wikitext
- Development notes
- Extensions used on Wikimedia
- VisualEditor
- Libraries