Extension:ParserPlayground

What it does
The Parser Playground is a harness for in-progress experimental work for the Future-facing parser & visual editor projects.

Now available
Currently (as of early July 2011) the playground holds:
 * integration into WikiEditor edit box/toolbar
 * ability to switch between modes, retaining data:
 * plain source wikitext
 * preview using MediaWiki's current parser
 * editable preview using experimental PEG-based parser (very incomplete)
 * toggleable parse tree inspector
 * When using the PEG-based parser, a primitive editing mode is provided which pops up a dialog for any chunk of the document you click on. You can edit the reconstructed source, which is then re-parsed and reinserted.

Coming soon
These should be going by the end of July:
 * settings controls to make it easier to plug in another parser variant
 * more fixes to round-tripping and more data types for PegParser
 * initial template expansion in the preview/editor
 * round-tripping batch tester
 * hand a page list or category over and get a report on what does/doesn't cleanly trip through the parser and back to source

Coming later

 * full implementation of Future/Parser plan API/structure & extension support
 * CLI test suite (using node.js)
 * more automated & semi-automated tests
 * integration of the 'wikidom' editing surface demos
 * integration of actual editing surface thingy

Gadget
On mediawiki.org, simply enable the 'Parser Playground' gadget on the gadgets tab of Special:Preferences.

Ensure that you have the new editing toolbar on; you may also have to explicitly enable the link dialogs to get the editing test to work.

You can copy the gadget to a local wiki (1.17 or later) via Special:Gadgets/export/ParserPlayground.

Extension
Currently built for compat on 1.19 or later, might work on 1.17/1.18.

Requires Extension:Gadgets and Extension:WikiEditor.

Screenshots
Traditional source editing looks something like this:


 * ParserPlayground-plain-edit.png

To start fiddling with ParserPlayground's experiments, open up the 'Rich editor' tab on the edit toolbar and switch the mode:


 * ParserPlayground-select.png

The mode you select will be remembered in a cookie.

'Source' you understand of course! Select this to switch back to regular editing when you're done.

'MediaWiki Parser' mode uses MediaWiki's current built-in parser to do a preview render; you can't edit in this mode, but you can see the rendered output, and enabling the inspector will show the XML preprocessor tree.

'PegParser' is an in-progress experimental parser built using a Parsing Expression Grammar definition and some custom code. This is far from complete, and still has some round-tripping problems so be sure to check changes before you save.


 * ParserPlayground-peg-inspector.png

For the PegParser mode, toggling the inspector can also highlight the association between a node in the rendered view and in the intermediate tree. Neat!


 * ParserPlayground-peg-edit-dialog.png

More interestingly, you can click on a node (a link, paragraph, template invocation, etc) to pop up a primitive editing box. This re-builds wikitext source from the parse tree, then on saving re-parses it and slips the updated text into the parse tree. There will be cases where this isn't entirely correct, but for now it's a quick way to try stuffing stuff in.


 * ParserPlayground-peg-edit-post.png

Inspector view layout
Basic editor:

Top/bottom like Firebug or classic WordPerfect "show codes" mode (currently using thise mode)

Sidebar (considering this?)

Code layout
This isn't super clean yet, but should pan out in a bit.

Core logic & integration with WikiEditor:
 * ext.parserPlayground.js & .css

Broadly there are a few separate chunks:
 * parsing to produce an AST/DOM from source
 * ext.parserPlayground.pegParser.js & pegParser.pegjs.txt - in-progress experimental parser building a hierarchical JSON AST/DOM
 * ext.parserPlayground.classicParser.js - uses MediaWiki API to create XML preprocessor tree
 * post-processing on the AST/DOM structure to expand templates, reassemble things, etc
 * ext.parserPlayground.pegParser.js - stubbed for now
 * serializer: round-trips the AST back to source
 * ext.parserPlayground.serializer.js - handles the JSON AST/DOM
 * ext.parserPlayground.classicParser.js - uses MediaWiki API to round-trip source from XML preprocessor tree
 * rendering to some sort of view
 * ext.parserPlayground.renderer.js - quick and dirty HTML output from the JSON AST/DOM, used for the PegParser preview/edit view
 * ext.parserPlayground.classicParser.js - uses MediaWiki API to render a preview from source

The parsing & post-processing bits will evolve into the Future/Parser plan basic layout for the to-come parser stuff, and you will be able to drop in an alternate class to implement the parser, as well as plugins to handle metadata detection & rendering for tag hook and parser function extensions.

Other code bits:
 * jquery.nodetree.js / .css -- jQuery plugin to create collapsable node tree views from XML or JSON data structures, used for the inspector
 * ext.parserPlayground.hashMap.js -- primitive HashMap class used for the inspector view to tie nodes in both panes
 * lib.jsdiff.js -- a simple inline diff library, currently unused
 * lib.pegjs.js -- PEG.js parser generator (PEG.js home)

Custom PEG grammar
You can set a global JS var  to the name of a local wiki page, which will be loaded instead of the master copy when selecting PegParser mode.

Since you're not replacing the renderer or serializer here, you need to stick with the existing AST structure.

Custom parser class
todo ... should be possible to whip one up based on current code but it's not friendly yet. :)

Parser extensions
todo ... will use the interfaces to be defined via Future/Parser plan etc.

Parser & tag hook extensions basically need to provide:
 * name of functions/tags
 * expansion-time callback to take input & context and return an expanded AST
 * if necessary, renderer callbacks to make something nice visible

Editor extensions
Plugins to provide special handling of particular things -- like a link editor, or an embedded code editor for syntax-highlighting source chunks, or something to edit SVG graphics inline, or a table editor, or .... will also be needed. There's not yet an interface ready.

Test extensions
More automated, semi-automated, and manual tests will be integrated in; ways to add to that should come along with.