Alternative parsers

Jump to: navigation, search

This page is a compilation of links, descriptions, and status reports of the various alternative MediaWiki parsers—that is, programs and projects, other than MediaWiki itself, which are able or intended to translate MediaWiki's text markup syntax into something else. Some of these have quite narrow purposes, others are possible contenders for replacing the somewhat labyrinthine code that currently drives MediaWiki itself.

Many of the things linked here are likely to be out of date and under-maintained, even abandoned. But in the interest of not duplicating the same work over and over, it seemed sensible to collect together what was "out there".

Known implementations[edit]

Name and link Principal author(s) Language Input Output Complete implementation Comments / other info License Max Freedom .NET Wiki title HTML No A website that converts Wiki markup to HTML. Allows user to browse for a Wiki title and return the full HTML or an abstract.
Wiky.php Toni Lähdekorpi PHP, Regular Expressions Markup HTML No A tiny PHP library that uses only regular expressions to convert Wiki markup to HTML. Apache License/GPL/LGPL/MPL/CC
sanskritnlp Vishvas Vasuki Scala Mediawiki text Mediawiki text and Section tree No Only parses mediawiki sections - that's it. One can parse a wiki page with multiple sections, get a section tree, add, access and delete sections. Creative commons
Wiky Tanin Na Nakorn Ruby Markup HTML No A simple Ruby library to convert Wiki markup to HTML. Apache License
Wiky.js Tanin Na Nakorn Javascript Markup HTML No A simple Javascript library to convert Wiki markup to HTML (limited subset). Apache License
txtwiki.js Joao Sa Javascript Markup Text No A javascript library to convert MediaWiki markup to plaintext. MIT License
wikipedia-js kenshiro_o Node.js Markup HTML No A simple client that enables you to query Wikipedia articles in english. The results are formatted in basic HTML. You can retrieve either a summary of an article (i.e. before the table of contents) or a full article. MIT
WikiExtractor Giuseppe Attardi, Antonio Fuschetto Python XML dumps text No Simple and fast tool for extracting plain text from Wikipedia dumps. It performs template expansion and handles parser functions (core and extended). GPL
mw2html Connelly Barnes Python Wiki url HTML No Minimal setup - gets the basic job of creating a static copy of the wiki done. Public Domain
mwlib Python with C library Markup and other parse tree, HTML, PDF, XML, OpenDocument No Part of cooperation between Wikimedia Foundation and PediaPress. BSD
Mediawiki2HTML Machine Johannes Buchner PHP Markup HTML No Project for parsing without the Mediawiki engine. AGPL3 + any later version
PHP5 WP Dan Goldsmith PHP Markup HTML No Parser With Plugin Framework To Add Additional Syntax. Configurable for alternative markup i.e. PMWIKI. MPL 2.0
Mylyn WikiText David Green Java Local files HTML, DocBook, Eclipse Help, DITA, extensible No Integration with Ant and Eclipse runtime.
Java API (Bliki engine) axelclk Java Markup fragment HTML, PDF almost Java Wikipedia API - (supports ParserFunctions, Lua/Scribunto...).
FlexBisonParse Timwi flex, bison and C Markup fragment Custom XML No Intended as an eventual replacement to the parsing code inside MediaWiki itself.
JAMWiki Ryan Java JAMWiki front-end HTML No Java Wiki engine that supports MediaWiki syntax. The roadmap also calls for XML import and export that will be compatible with Mediawiki.
InstaView Pilaf JavaScript Markup fragment HTML No Provides instant preview while editing a page (without reloading).
InstaView C. Scott Ananian JavaScript Markup fragment HTML No Port of Pilaf's code to node.js, volo, and the browser.
Perl Wikipedia Toolkit Michal Jurosz Perl XML dump, SQL dump Own parse tree, WikiMedia markup No Perl Wikipedia Toolkit developed for Computer-assisted Wikipedia translation. (Little functional)
Text_Wiki_Mediawiki Multiple PHP Markup HTML, Latex, Plain text No Part of the Text_Wiki library.
TomeRaider export Erik Zachte Perl XML dump TomeRaider database No See en:Wikipedia:TomeRaider database for more details.
Waikiki Magnus Manske C++ SQL dump (via SQLite) HTML No Abandoned in favour of "flexbisonparse", but has been used inside some experimental "front ends".
Wikiwyg Jim Higson JavaScript A live installation of MediaWiki HTML (via XML) No More than just a parser; attempts to create a fully functional client-side interface.
wik2dict Guaka Python SQL dump DICT No
wiki2pdf Stephan Walter Python (and PHP) Markup fragment or set of online articles LaTeX, PDF No Project is incomplete and dormant.
wb2pdf Dirk Hünniger Haskell online article LaTeX, PDF, Parse Tree No Recursive Descent based on Monadic Parser Combinators. Allows for non context-free input, especially non well formatted HTML as often found on Wikipedia. GPL
WikiPDF Felipe Sanches Python (and PHP) One selected article LaTeX based on templates, PDF No Mediawiki extension that uses Stephan Walter's wiki2pdf as backend.
Wiki2XML Magnus Manske C++ Markup fragment (?) Custom XML No Another aborted project on the way to 'flexbisonparse'.
HTML2FPDF Renato A. C. PHP A PHP class that transforms HTML into a feed for FPDF resulting in a PDF file HTML -> HTML2FPDF -> FPDF -> PDF No Not specifically for Mediawiki, but easy to install using an updated version of this tool:updated html2fpdf.php. See HTML2FPDF and Mediawiki for more instructions.
WikiOnCD Andrew Rodland Perl SQL Dump or markup HTML, Parse tree (eventually?) No Started out as an offline wiki browser, but grew a parser when Wiki2static turned out to be too limiting. No web presence yet; code is in the SVN. GPL
WikiTaxi Ralf Junker Delphi / Pascal MediaWiki markup, page or fragment Node-tree, HTML, potentially others almost Hand-crafted parser with template expansion, parser functions (core and extended), tag extensions (<ref>, <source>), wiki text parsing. Used for the WikiTaxi offline reader. No sources available
Wikifilter  ? C++ (VS) XML dumps HTML No A Windows program that uses Apache/IIS to serve the pages. Abandoned in 2006, before ParserFunctions were available.
Wikipedia Dump Reader Benjamin Thyreau Python XML dumps On screen No Cross platform viewer. GPLv2/~BSD license
Marker Ryan Blue ruby Markup (subset) HTML or formatted text No Marker is a ruby implementation of a subset of the MediaWiki markup language, intended bring MediaWiki's markup language to non-wiki applications with multiple output formats. GPL
WikiCloth nricciar ruby Markup HTML No Ruby implementation of the MediaWiki markup language, including a fair amount of the parser functions. MIT
XWiki XWiki dev team Java Various WikiMarkups Well formed sequence of events, HTML/XHTML, other WikiMarkups No XWiki can be used a full-fledged wiki supporting several WikiMarkups (including MediaWiki's markup). It also offers a standalone Rendering Engine that can be used as a Java library for parsing/rendering WikiMarkups. Cant output to mediawiki format as of 2016/03 though. LGPL
Kiwi Thomas Luce, Karl Matthias, C, Ruby, PEG Markup HTML almost Kiwi is a PEG-based C implementation with Ruby bindings and a command line parser. It is very fast and supports most of the MediaWiki syntax. Actively developed. BSD
YaCy YaCy dev team Java XML Dump XML with Dublin Core Metadata No YaCy is a search engine and a MediaWiki parser is included as one of the import modules. MediaWiki xml dumps are first converted to Dublin Core XML as intermediate format and then inserted into the search index using the built-in Dublin Core importer. GPL
MessageParser Neil Kandalgaonkar JavaScript Markup Abstract syntax tree, jQuery object, HTML No Designed for use with message strings, to allow enhanced interface in the browser, like pluralizing internationalized messages or attaching jQuery behavior to links within a message. GPL
Sweble Wikitext Parser Hannes Dohrn Java Markup Abstract syntax tree, XML, HTML almost Claims to be very thorough. Apache License 2.0
JWPL api Torsten Zesch, Richard Eckart de Castilho, Oliver Ferschke, Elisabeth Niemann Java XML Dump API to access pages, outlinks, inlinks and more No "JWPL (Java Wikipedia Library) is a free, Java-based application programming interface that allows to access all information contained in Wikipedia." "JWPL is for you: If you need structured access to Wikipedia in Java." Older parser not maintained any more - JWPL uses Sweble now. LGPL
libmwparser Saitmoh C XML dumps, Markup XML, XHTML, Expanded WikiText almost Primary an wikimedias offline reader with interwiki support. Libmwparser is a source independent library which supports most of MediaWiki syntax and some extensions like math or gallery. GPL
mediawiki-parser Peter Potrowl
Erik Rose
Python Markup XHTML, raw text, AST No GSoC-2011 project; the use of a PEG parser makes it easy to improve.
Parser functions are not supported yet.
Parsoid Gabriel Wicke and the Parsoid / Visual editor team PEG / JavaScript / Node.js Markup, XML dumps, test cases Tokens, HTML5 DOM with RDFa and round-trip data Yes Fully-featured round-tripping parser/runtime that powers the Visual editor on Wikipedia. Work ongoing to provide a HTML-only read / edit interface, and later to become the default parser for MediaWiki. See roadmap. Used to make this edit. GPL
mwparserfromhell The Earwig Python Markup AST No A Python library to convert Wiki markup to a navigable string, which can be used to examine and manipulate templates. Written in pure Python, compatible with Python 2.7 and 3, and no dependencies. MIT License
Saya.Parser.Wiki Nana Sakisaka C++ Markup Abstract syntax tree No Pure C++11 parser implemented with Boost.Spirit.Qi. Boost Software License 1.0 Marcus Brinkmann Python Markup AST, HTML No Stateful PEG parser based on Grako, with a very clean separation of parsing stages, grammars and semantic transformations. BSD
Pandoc John MacFarlane Haskell Markup many No Can convert subset of mediawiki markup to ~35 different formats (5 of which are flavors of markdown). GPLv2
Wikiforia Marcus Klang Java XML Dumps, Markup Text No Uses the AST output from Sweble Wikitext Parser internally to produce raw text. Can parallel decompress and parse compressed multistreamed xml dumps. GPLv2
wtf_wikipedia Spencer Kelly Javascript Markup JSON almost Supports recursive links & templates, parses infoboxes and links, resolves special templates, parses images and categories. runs server-side & browser. MIT
Wiki-infobox-parser Zhipeng Jiang JavaScript Markup JSON No A light Wikipedia Infobox Parser written in JavaScript. MIT
wikitextparser 5j9 Python Markup AST almost Provides several accessor methods in an object tree to navigate to structural elements like sections, tables, links etc. Supports extracting table data as list of lists. Available via pip, supports Python 3. GPL
PHP-Wikipedia-Syntax-Parser Don Wilson PHP Markup Associative array No Given raw contents and title of a Wikipedia article, this will output highly useful information in an organized fashion.
MwParserFromScratch CXuesong C# Markup AST No A portable .NET library that parses wikitext into Abstract Syntax Tree. For now it supports most of the common markup expressions except file links, double-underscored magic words, and tables. Apache License
gensim.segment_wiki RaRe Technologies Python MediaWiki XML JSON almost Gensim is a robust open-source vector space modeling and topic modeling toolkit implemented in Python, segment_wiki - script for wikipedia parsing & extraction. LGPLv2.1

A non-parser dumper[edit]

One of the common uses of alternative parsers is to dump wiki content into static form, such as HTML or PDF. Tim Starling has written a script which isn't a parser, but uses the MediaWiki internal code to dump an entire wiki to HTML, from the command-line. See Extension:DumpHTML. This has been used (years ago) to create the static dumps at

There are also similar dumpers as part of the Kiwix project, for example mwoffliner, and you can query the RESTBase API to obtain HTML-format output with semantic information (such as tranclusions) included.

Related topics[edit]