Parsoid/Extension API

Raw thought dump about how extensions interface with Parsoid.


 * In this first pass, we are only looking at extensions that implement implement tag handlers. We have support for extensions that implement content handlers as well and we will update this page.
 * In the Parsoid world, extensions will NOT get direct access to the wikitext engine. All interaction happens through an API and hooks (primarily transformation hooks and less pipeline event hooks). Ex: init, toDOM, fromDOM, postProcessDOM.
 * Extensions will interact with Parsoid via wikitext strings and DOM trees, i.e. they will convert wikitext strings to DOM trees or vice versa, or transform DOM trees in place.
 * Extensions should not expect to maintain global document state within the extension where ordering matters. Parsoid does not guarantee the order in which repeated occurrences of the same extension tag will match the order in which they are seen on the page (for ex: because of concurrent / asynchronous execution). Nor should extensions assume that they will be invoked for every instance that is seen in wikitext (for ex: because we reuse parsed content from a cache). So, this means, global state like counters cannot be reliably maintained by the extension. Extensions can get access to the fully processed DOM of the page which they could inspect to reconstruct ordering.
 * Since Parsoid transforms wikitext to HTML and HTML to wikitext, extensions need to be aware of the HTML to wikitext transformation. Parsoid provides a basic default HTML to wikitext transformation based on information encoded in the data-mw attribute during the wikitext to HTML transformation. However, if extensions intend to provide custom editing support in editing clients like Visual Editor, they should provide handlers that can inspect edited HTML for their extensions and convert it back to appropriate wikitext.

Extension registration and configuration
Parsoid will use the same extension registration interface that core uses. However, Parsoid will only recognize those extensions that implement the  interface. Currently, this interface has exactly one method:. The config object is an associate array with the following fields currently:


 * : The name of the extension
 * : If an extension implements extension tags (ex: Cite implements and :  Style modules that this extensions exports and need to be included in the list of modules on the page.
 * FIXME: Should this be a per-extension-tag configuration, vs a per-extension configuration?
 * FIXME: Should this be a more generic modules property vs. being a styles property?
 * : If an extension needs to inspect the global document after is it is constructed, extensions are expected to register a DOM processor that is invoked by Parsoid. This DOM processor is expected to transform the DOM in place as appropriate. To future proof against new DOM transformations that Parsoid might support for extensions, domProcessors is an associative array that maps a transformation to an implementation class that implements an interface. Currently, only the  transformation is supported.

ExtensionTag abstract class
Implementations of extension tags should extend the  abstract class. This class provides four methods:,  ,  ,. Extensions are only expected to implement the toDOM method at the very least (otherwise, what is this extension tag even doing?). Parsoid takes care of annotating the output DOM fragment returned by the toDOM method so that it can be appropriately converted back to wikitext. Please look at the docs for this class for more specific details about these methods.

Configuring extension tags
The extension tag config object is an associative array with the following fields currently:


 * : The name of the tag
 * : The class that provides the implementation of this tag. This class should extend the  abstract class as above
 * : This options block dictates how the DOM fragment returned by the  method should be handled. Currently, only one option exists. The vast majority of extensions will not need this.
 * : By default, Parsoid takes the DOM fragment returned by the  method and splices it into the parent document in the appropriate place. However, if   is , Parsoid will instead leave a marker instead and store the fragment in a map. It is expected that the extension's   DOM processor will appropriately deal with these DOM fragments and manipulate them. For example, the Cite extension relies on this to migrate the ref's fragments to the references section and leave behind a citation that is appropriately globally numbered.
 * : This options block influences Parsoid's HTML to wikitext transformation. Given that extensions might implement their own  implementation, these options primarily influence how the generated wikitext interacts with its context. Currently, only one option exists. FIXME: Should this be called   instead?
 * : By default, the wikitext from converting the HTML is rendered inline. However, if extensions specify a  value for this property, the wikitext output is rendered on its own separate line.

Parsoid API for extensions
As part of implementing the various methods (toDOM, fromDOM, etc) for extension tags and the DOM post processors, extensions might need access to certain kinds of information or functionality. For example, extensions that intend to handle wikitext as part of their implementation will rely on Parsoid to convert that wikitext to HTML. Or, they might need access to configuration information for the wiki, or the page. Or, they might need to log error messages or metrics. The  class provides this API. Please look at the docs for this class for specific details about the interface. The following sections document the API methods in a few broad classes with some discussion of how / where to use them.

Converting wikitext to DOM
Extensions that used  when interacting with the MediaWiki core parser have two different methods to choose from:


 * transforms wikitext to a DOM tree. Extension tags that have special wikitext semantics or might have embedded wikitext snippets should use this method. For example, gallery extension has images whose wikitext are processed separately per-image and put together in a div block.
 * transforms an extension tag to a DOM tree rooted in a requested wrapper tag (ex: div, span, sup). Extension tags that wrap wikitext (ex: ref, references, poem) should use this method. Internally, this method calls  which is a more general purpose method.

The wikitext passed in to these methods are processed fully - there is no notion of partially processed wikitext in Parsoid. The following options are provided which are of declarative / semantic nature. Beyond this, extensions will not be able to turn on / off specific pieces of the parsing pipeline. In the long run, this makes for simpler semantics and more robust code since (a) the underlying implementation can be changed without breaking extensions (b) wikitext doesn't behave differently when used outside extensions and inside extensions which makes for a better user experience.


 * : This specifies that the output of this wikitext will be embedded in an inline / phrasing HTML context. This effectively turns off paragraphs and pre behavior.
 * : This is a special case of the  option and is a backward compatibility hack for what the MediaWiki core parser assumes. We are working to either get rid of this or split out the difference into its own option, if found necessary.

Converting HTML to DOM and DOM to HTML

 * : Where extensions need to parse HTML and construct a DOM document (for example, creating an empty base document, or for processing HTML snippets), they should use the  method since Parsoid does some additional data management of the DOM with performance in mind. All of Parsoid core code assumes this canonical DOM data representation and without that, extensions might experiences subtle (or not so subtle) failures in certain scenarios.
 * : For the same reason as above, where extensions need to serialize a DOM node to string, they should use the  method which knows about the internal data representation while serializing. After using this method, the input DOM is no longer in Parsoid-canonical form. So, extensions should not expect to continue using the input DOM with Parsoid's API after this method is called.
 * : The output HTML string of this method will be identical to that of the  method above. However, unlike , extensions can continue using the input DOM fragment after serialization. It is safe to always use this method everywhere instead of  , but for larger DOM fragments,   is a tad more efficient since it doesn't do additional work to put the DOM back in canonical form.

Sanitization helpers
Parsoid discourages extensions from using the Sanitizer class directly at this time. The API currently escapes a few sanitization helpers based on current usage and analysis. The helpers handle extension arguments, CSS, HTML ids, as well as attributes for a HTML element

Converting DOM to wikitext
This part of the API is only relevant to extensions that intend to provide custom editing support for their extensions in editing clients like VisualEditor. For example, the Cite and Gallery extensions make use of this API. The methods in this section mirror those in the wikitext to DOM section.


 * : All extension tags will need to use this. This method takes care of converting the HTML attributes to the extension's arguments while handling Parsoid-specific annotations.
 * : Extensions that used the  method will use this method to convert their input DOM to wikitext. This is for scenarios where the wikitext embedded in the extension tag has no special semantics.
 * : Extensions that used the  method will use this method to convert DOM fragments to wikitext. This is for scenarios where the extension either has special semantics for its wikitext OR has embedded wikitext as part of its extensions input. So, naturally, the extension will have to tease out those relevant DOM fragments from its input DOM and convert those fragments to wikitext.
 * : For these extensions that manage their own content and DOM, they may have scenarios where they may need to escape wikitext-like constructs in a string so that the string can be used as part of a larger wikitext fragment without breaking those semantics.. For example, wikitext used in template arguments cannot use the  as is, wikitext used in table cells cannot use   as is, and so on. This method lets extensions delegate this logic to Parsoid and provides a few pre-defined context options currently and this will be expanded in the future based on usage and further analysis. Note that extensions that rely on Parsoid for their entire embedded wikitext will not have to deal with it.   handles this automatically on their behalf.

... to be completed ...