Markup spec

MediaWiki markup spec project:

Goals

 * Produce a specification of MediaWiki's markup format that is sufficiently complete and consistent that multiple compatible parser implementations can be built from it.
 * Spec may or may not use EBNF etc. Might have to just use lots of words. ;) ANTLR is looking very promising- see /ANTLR.
 * Define a data model for a parse tree
 * The data model should be representable in XML, though an official XML schema for such a representation may or may not be defined.
 * Round-trip conversion between source code and the data model must be possible. There may be a many-to-one relationship between source code and parse trees, but the canonical transformation from parse tree to source code should always parse back to the same parse tree.
 * A parser built from this spec will replace MediaWiki's current parser in the future.
 * This will potentially allow lots of nifty new features that are currently either Not Possible or Very Hard, e.g. WYSIWYG editing.

Compatibility

 * In general, the spec will strive to avoid deviating from present behavior where it is reasonable and well-defined, and will seek to avoid adding new behaviour without considering whether it may break already existent pages.
 * Where the current parser's behavior is undefined or obviously buggy, the spec may define new behavior which is different.

Difficulties

 * Some of the syntax is kind of hairy. Bleah!
 * Language-sensitive and otherwise customizable keywords.
 * Extensions...
 * Integrated HTML and HTML-like tags.
 * Lots of scary context-sensitivity (see e.g. MediaWiki lexer).

Resources

 * Raid Magnus's wiki2xml work for some starting points; examine how his parser works (and how it differs from the main one) and the intermediate XML format he uses
 * User:HappyDog/WikiText parsing - some observations based on 1.3.10 by HappyDog
 * meta:Help:Editing - it's a start
 * An attempt to describe the markup in BNF form: Markup spec/BNF
 * http://jamwiki.org/wiki/en/StartingPoints Aims to have Mediawiki compatible syntax see http://svn.sourceforge.net/viewvc/jamwiki/wiki/trunk/src/lex/ for an attempt to write a parser.
 * http://mediacloth.rubyforge.org Is a Ruby library for parsing Mediawiki compatible syntax to XHTML.
 * http://mediacloth.rubyforge.org Is a Ruby library for parsing Mediawiki compatible syntax to XHTML.


 * Tim Starling's MediaWiki preprocessor syntax in ABNF (RFC 5234)


 * Wikitext-l maillist: wikitext-l

The Markup Language
The MediaWiki markup language (commonly referred to within the MediaWiki community as wikitext, though this usage is ambiguous within the larger wiki community) uses sometimes paired non-textual ASCII characters to indicate to the parser how the editor wishes an item or section of text to be displayed. The parser translates these tokens into (X)HTML as closely as semantically possible.

v1.6 markup tokens
The markup tokens fall into two broad categories: unary tokens (like : or * used at the beginning of a line), which stand alone, and binary tokens (like those for italic or boldface) which must be used in matched pairs. Unary tokens may only be preceded by comments or whitespace; otherwise, they will not be interpreted.

Start of line only

 * blank line: paragraph break (HTML &lt;p&gt;)
 * Horizontal line:  (4 or more hyphens), specified in /BNF/Article
 * Pre-formatted text: (space)
 * Lists
 * Bulleted: *
 * Numbered: #
 * Indent with no marking: :
 * Definition list: ;
 * Notes:
 * These may be combined at the start of the line to create nested lists, e.g. *** to give a bulleted list three levels deep, or **# to have a numbered list within two-levels of bulleted list nesting.
 * Redirects: #redirect or #REDIRECT (followed by wikilink)
 * The whole quagmire that is table formatting:  {| ... |}  with in between  |- |+ || | ! ! .

Can be used anywhere

 * "Magic words", e.g.  ,    (see Help:Magic words)
 * Signatures:
 *   Replaced with your username
 *  ~  Replaced with your username and the date
 *   Replaced with the date.
 * Notes:
 * These tags are replaced at the point the edit is saved.
 * Magic links: ISBN ..., RFC ..., PMID ... (see /BNF/Magic links/)

Binary
The ellipses (...) are used to indicate where the content goes and are not part of the markup.

Beginning of a line

 * Equals signs are used for headings (must be at start of line)
 * 1st level heading:  = ... = 
 * 2nd level heading:  == ... == 
 * 3rd level heading:  === ... === 
 * 4th level heading:  ==== ... ==== 
 * 5th level heading:  ===== ... ===== 
 * 6th level heading:  ====== ... ====== 
 * Specified in /BNF/Article

Anywhere

 * Square brackets are used for links:
 * Internal/interwiki link + language links + category links + images:  ...   (see also Namespaces below)
 * vertical bars separate optional parameters, which are:
 * link: first parameter: display text (also defaulted using "pipe trick") (also trailing concatenated text included in display, e.g. s for plural)
 * image: many parameters; see w:Wikipedia:Extended image syntax
 * category: first parameter: sort order in category list
 * link contents have to be parsed for whether they're dates if $wgUseDynamicDates is on
 * External link:  [ ... ] 
 * space separates optional first parameter, which is display text
 * undecorated URLs are also recognized and hotlinked
 * Specified in /BNF/Links
 * Apostrophes are used for formatting:
 * Italic:   ...  
 * Bold:   ...  
 * Bold + Italic:   ...  
 * Note that improper nesting of bold and italics is currently permitted.
 * Curly braces are used for transclusion:
 * Include template:   (see also Namespaces below)
 * Unlimited number of optional pipe-delimited parameters, each of which may optionally start with a parameter name preceding an equals sign
 * Include template parameter:  
 * Optionally including a pipe followed by the parameter default: will use the first passed in parameter, and if none is received, will insert "Bob" instead.
 * Use a built-in variable:   (see m:Help:Variable)
 * Call a parser function:  
 * Various HTML style tags:
 * &lt;nowiki&gt; do not interpret wiki markup, do allow newline in list and indent elements (but still flow text, still allow SGML entities)
 * &lt;pre&gt; do not interpret wiki markup, do not flow text (but still allow SGML entities)
 * &lt;math&gt; if $wgUseTeX is set
 * &lt;html&gt; if $wgRawHtml is set
 * &lt;gallery&gt;</tt>
 * &lt;onlyinclude&gt;</tt> &lt;noinclude&gt;</tt> &lt;includeonly&gt;</tt>
 * Parser extension tags, like &lt;ref&gt;</tt> (using Cite.php)
 * Plus most 'non-dangerous' HTML tags: 'b', 'del', 'i', 'ins', 'u', 'font', 'big', 'small', 'sub', 'sup', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'cite', 'code', 'em', 's', 'strike', 'strong', 'tt', 'var', 'div', 'center', 'blockquote', 'ol', 'ul', 'dl', 'table', 'caption', 'pre', 'ruby', 'rt', 'rb' , 'rp', 'p', 'span', 'u', 'br', 'hr', 'li', 'dt', 'dd', 'td', 'th', 'tr'
 * &lt;!-- ... --&gt; HTML-style comments
 * SGML entities: &amp;...;</tt>

Namespaces
In wikilinks and template inclusions, colons set off namespaces and other modifiers:
 * proper namespaces: <TT>Talk:</TT>, <TT>User:</TT>, project, etc.
 * "special" namespaces: <TT>File:</TT> (was <TT>Image:</TT>), <TT>Category:</TT>, <TT>Template:</TT>
 * pseudo-namespaces: Special:</tt>, Media:</tt>
 * lone/leading :</tt>
 * lone :</tt> forces main namespace
 * leading :</tt> allows link to image page rather than inline image, or similarly to category or template page
 * interwiki links:
 * same project, different language: code of two or more letters
 * different project, same language: w:</tt> for Wikipedia, wikt:</tt> for Wiktionary, m:</tt> for Meta, etc. -- see m:Help:Interwiki_linking for more information (especially when using in templates; transwiki transclusion, iw_trans)
 * <tt>subst:</tt> force one-time template substitution upon edit, rather than dynamic expansion on each view
 * <tt>int:</tt>, <tt>msg:</tt>, <tt>msgnw:</tt>, <tt>raw:</tt> -- see m:Help:Magic words
 * <tt>MediaWiki:</tt> magically access mediawiki formatting and boilerplate text (e.g. MediaWiki:copyrightwarning)
 * Standard parser functions: <tt>UC:</tt>, <tt>LC:</tt>, etc. (see m:Help:Parser function)
 * Additional parser functions: <tt>#expr:</tt>, <tt>#if:</tt>, <tt>#switch:</tt>, etc.
 * other extensions?

Several combinations of the above are possible, e.g. m:Help:Variable -- help namespace within Meta project.

From MetaWiki
The following text was at Wikitext Metasyntax and needs to be merged in here.

Basic Markup
Define markups

Parser outline
Another way to check whether we've covered everything in the grammar is to look at the steps the parser actually goes through:


 * The preprocessor does:


 * 1) Strip (hooks before/after)
 * 2) Remove HTML-like comments <<<<!>!>!>!>
 * 3) Replace variables
 * 4) Subst
 * 5) MSG, MSGNW, RAW
 * 6) Parser functions
 * 7) Templates


 * The parser does:


 * 1) Strip (hooks before, after)
 * 2) treats nowiki, pre, math and possibly other with "userfunc tag hooks" hiero)
 * 3) Removes HTML-like comments
 * 4) *HTML comments are removed. (this text by HappyDog)
 * 5) *Any tags that are not allowed by the software (e.g. tags) are replaced by HTML entitities, so they display as literals and are not treated as HTML by the browser.
 * 6) *Any badly formed tags (e.g. nested tags that shouldn't be nested,  tags outside a   tag, etc.) are also replaced by HTML entitities so they are not treated as HTML.
 * 7) *Any attributes that are not allowed by the software (e.g. onMouseOver) are removed from otherwise valid tags.
 * 8) *A small amount of minor source formatting is applied (basically, the removal of unnecessary whitespace).
 * 9) *A closing tag is added at the end for all tags that are not closed properly. Note that some tags (e.g.   ) don't need to be closed.


 * 1) Internal parse
 * 2) Noinclude/onlyinclude/includeonly sections
 * 3) Remove HTML tags
 * 4) Replace variables
 * 5) Hooks: Internalparsebeforelinks
 * 6) Tables
 * 7) Magic words
 * 8) Strip TOC
 * 9) Strip no gallery ( __NOGALLERY__ )
 * 10) do headings
 * 11) Do dynamic dates
 * 12) Do quotes (  and ' )
 * 13) Replace internal links
 * 14) Process images (do the caption recursively as it might contain links, or even other images...)
 * 15) Process categories
 * 16) Replace external links
 * 17) Re-replace masked internal links
 * 18) Do magic links (ISBN, RFC...)
 * 19) Format headings ( __NEWSECTIONLINK__, ...)
 * 20) Unstrip general
 * 21) Fix tags (french spaces, guillemet)
 * 22) Blocks (lists etc)
 * 23) Replace link holders
 * 24) Language converter:
 * 25) Normal text converted on a word by word basis(?) if autoconvert is enabled
 * 26) Text in -{code1:text1;code2:text2;...}- blocks converted manually
 * 27) Text in -{...}- not converted at all.
 * 28) Unstrip no wiki
 * 29) Extra tags and params
 * 30) User funcs?
 * 31) Un strip general
 * 32) Normalise char references
 * 33) Tidy + hook


 * The save parser does:
 * 1) Convert newlines
 * 2) Strips
 * 3) Pass 2
 * 4) Substs
 * 5) Strip again? gallery something.
 * 6) Signatures
 * 7) Pipe tricks
 * 8) Trim trailing whitespace
 * 9) Unstrips

See also m:Help:Magic words.