Parsing/Replacing Tidy/FAQ

What is Tidy?
Tidy is a library currently used by MediaWiki to fix some HTML errors found in wiki pages. Badly formed markup is common on wiki pages when editors use HTML tags in templates and on the page itself. (Example: unclosed HTML tags, such as a   without a  , are common). In some cases, MediaWiki can generate erroneous HTML by itself.

Tidy fixes these markup errors, but also does other "cleanup" on its own that is not required for correctness. For example, it removes empty elements and adds whitespace between HTML tags, which can render>Parsing/Replacing Tidy#display:inline list wrapping diffs because of inter-element white-space .28possibly a concern in other rendering scenarios.3F.29|sometimes change rendering. Since Tidy is based on HTML4 semantics and the web has moved to HTML5, it also makes some incorrect changes to HTML to 'fix' things that used to not work; for example, Tidy will unexpectedly move a bullet list out of a table caption even though that's allowed.

Why are you replacing it? And with what?
Tidy's technology is from the 1990s, when browsers weren't standardized. Tidy's behavior is loosely based on HTML4 semantics but matches no modern browser. After spending years without active maintenance, Tidy has now been revived as "tidy-html5" with very different behavior. The older Tidy is no longer being packaged. As noted earlier, Tidy does HTML cleanup unrelated to fixing errors. Together, all these issues have led to lots of bugs filed against it on Phabricator, and a replacement has been asked for since at least 2013.

HTML5 is the standard today, and the parsing algorithm for HTML5 is clearly specified, which has led to compatible implementations across browsers and other libraries. This algorithm also clearly specifies how broken markup should be fixed up. In this new technological landscape, Tidy should really be replaced with a HTML5 parser that fixes up the broken markup and generates valid, well-formed HTML markup in the standard way.

However, Wikimedia wikis have a huge corpus of pages whose markup relies on Tidy's fixups. Doing an immediate and straight-forward replacement of Tidy with a third-party HTML5 based tool is not feasible, since a HTML5-based tool would repair some markup differently and this can break how pages look.

So, we are replacing Tidy with our own tool based on the HTML5 specification, but which also adds a few Tidy-compatibility workarounds to minimize the impact of replacing Tidy. After experimenting with 3 different solutions, we have settled on RemexHTML, a PHP-based HTML5 parser on top of which we have written Tidy-compatibility passes to replicate some Tidy behaviour which we need to provide for now. In the future, RemexHTML could also be used to enable new core MediaWiki features, such as more-robust section editing, balanced template support, and more efficient page updates after templates have been edited.

For those who are wondering, note that using tidy-html5 would not preclude us from having to deal with fixing markup errors, since some of the required cleanup is due to changing from HTML4 to HTML5 semantics. There are other change-management reasons for preferring a in-house tool, including the ability to enable other features as mentioned above.

If you're interested in other technical details, please see https://phabricator.wikimedia.org/T89331 or byetidy>Parsing/Replacing Tidy|Replacing Tidy.

Which tests have you performed so far?
To identify the impact of replacing Tidy with a HTML5 based tool, we have utilized a testing strategy (using a tool called "VisualDiff") that compares the pixel-by-pixel output image of MediaWiki with Tidy enabled, with the pixel-by-pixel output image of its replacement. Early on, we found that a common difference was minor vertical whitespace changes. In the belief that these would either not be noticeable or would be tolerable, we wrote a tool called "UprightDiff" which is able to identify vertical motion within an image and to discount such motion for the purposes of automated testing. This also let us assign a numeric score to differences and readily identify the most egregious differences.

We exported a subset of about 64,000 articles (some from the recent changes stream, and rest selected randomly) from various Wikimedia wikis (40 wikis from Wikipedia, Wikisource, Wiktionary, and Wikivoyage), and rendered them with Tidy and with RemexHTML, then used "UprightDiff" to analyse the result. This takes a lot of cpu cycles, memory, and disk space, and it takes 2 days for one round of testing to complete. This limits the size of the testing corpus but we believe 64K is a sizeable sample to figure out the kind of fixes necessary.

To minimize the differences and reduce the impact of fixes that would be needed from editors, we added some additional Tidy-compatibility fixups. Since we found that self-closing tags were extremely common on wikimedia wikis, we added a compatibility fix to treat them as empty tags (i.e.    is treated as   ). We added some other compatibility passes as well. After all these fixes were in place and we repeated our tests, we found that 93.4% of pages had no changes in rendering. And, 96.9% of pages had either no pixel diffs (93.4%) or insignificant vertical whitespace shifts only (3.5% = 96.9 - 93.4). The remaining 3.1% pages (100 - 96.9) showed pixel differences that had other reasons.

Based on these tests, we identified several classes of markup errors that will render differently between the two. For one class of markup errors (self-closing tags that aren't valid in HTML5), we added a wdata>d:Q25885956|maintenance category that editors have already been using to fix up templates and pages. But, the other classes of markup errors are not easy to detect automatically at this time and editors' assistance is necessary to identify and fix them up.

What will happen? When will these changes happen?
As noted earlier, we cannot yet do a drop-in replacement of Tidy with a HTML5-based tool.

We have added a maintenance category for one class of markup errors, which will help editors fix them up. To help editors identify and fix up other markup errors, we also built a ParserMigration extension that helps them compare output in production and fix markup errors.

Separately, we have also built the Linter extension to to identify some of the fixes that are needed.

As of of end March 2017, we deployed the ParserMigration extensions to all wikis.

The Linter extension is currently deployed on small and medium wikis.

We hope to have Linter deployed to large wikis by mid May. Via these extensions, we hope to enable editors fix up pages for actually replacing Tidy in 2017. Once enough fixes are made and we are satisfied that the impact on editors and readers will be minor and tolerable, we will go ahead and replace Tidy But, we would also not like to drag this on indefinitely.

So, it would be ideal if the help>Special:MyLanguage/Help:Extension:Linter#Goal: Replacing Tidy|high-priority issues identified by the Linter extension are prioritized by editors.

Separately, the Tidy-compatibility fixups (mentioned in the previous section) are meant to be in place till we replace Tidy. After that, we will start replacing these fixups gradually while relying on similar testing and tooling support.

What will editors need to do?

 * We anticipate that certain templates will need to be updated. testnotes>Parsing/Replacing Tidy#Test result notes|Test results from visual diffing has some examples of templates that need to be fixed. For one class of markup errors, we expect the Linter extension (that surfaces information from Parsoid) will greatly help in identifying templates that might be source of markup errors. We don't yet know how to automatically determine all affected templates, but we expect that the list of templates and suggestions will get more complete as we learn about problems that were not caught before. Community Liaisons will reach out to template experts, regulars at technical village pumps, etc. to make sure they become aware of the necessary changes and of resources that may help them.
 * As mentioned above, to assist editors in migrating wikitext to the new rules, we have deployed the Linter and ParserMigration extensions. If you enable "ParserMigration" in your preferences, a link will be added to the toolbox of all articles, which can show the current (Tidy) and expected (RemexHTML) output side-by-side. You can preview article text changes in the same side-by-side view and see how your edits changes / fixes rendering.
 * We hope to have the Linter extension enabled on all wikis by mid-May 2017. As indicated on the help page, this identify wikitext patterns in pages and templates that need to be fixed.
 * We are thinking about other easy, sustainable ways in which we may support "gnomes" and all the Wikimedians willing to help with this effort.

Other FAQs
Q: What happens to <tvar|preslash> </> and similar tags if they are used in self-closed form?

A: As noted in T134423, the only valid self-closed HTML tags are: <tvar|area> </>, <tvar|base> </>, <tvar|br> </>, <tvar|col> </>, <tvar|embed> </>, <tvar|hr> </>, <tvar|img> </>, <tvar|input> </>, <tvar|keygen> </>, <tvar|link> </>, <tvar|meta> </>, <tvar|param> </>, <tvar|source> </>, <tvar|track> </>, <tvar|wbr> </>. Non-HTML self-closing tags (like <tvar|references> </> and <tvar|funnynowiki> </>) are not affected by this change. <tvar|pre> </> is a special case because while it is a HTML tag, MediaWiki treats it like an extension tag and hence remains unaffected. All other self-closing HTML tags should be fixed (and are already being fixed by editors at this time).

Since this usage is found in a lot of pages in our testing, in order to prevent unexpected rendering effects (e.g. <tvar|bslash> </> being treated as <tvar|bold> </> and causing more text being bolded than intended), we added a fix to the parser to convert them to an empty tag (eg. <tvar|bslash> </> will be converted to <tvar|bothbs> </>). But, we don't intend to retain this fix indefinitely. So, we would like for editors to continue fixing this deprecated usage.

Q: What were the results of tests on languages other than English, or on sister projects?

A: There is nothing in what Tidy, RemexHTML do that is specific to Wikipedia or English. This project is primarily about a change from HTML4 to HTML5 semantics and getting rid of some Tidy cleanups of HTML. These changes affects all projects and languages equally, except if some projects and languages tend to have more markup errors or use more self-closed tags than others.

Q: What other changes are editors likely to see, after this replacement?

A: The effect of this replacement is primarily going to affect readers, as they may notice that the page doesn't look right (for example, excessively wide navboxes without line breaks or wrapping). However, if anything, this might lead to the rendering seen in VisualEditor to match the rendering seen outside it much more than before, since Parsoid's output has been HTML5-compliant since the beginning, and we are now moving the read output to HTML5. We do not expect any impact on VisualEditor edits, but we will promptly address any bugs reported with respect to dirty diffs. In addition, we do not plan to add any error messages or warnings displayed on pages if the markup errors are not fixed.

Q: How does the replacement relate to other projects you are working on?

A: By enabling the move to HTML5 semantics, this is one of the steps evolving markup in our corpus to keep up with web standard. We also expect to leverage this tool to support well-balanced template output. Separately, but relatedly, this will also make the output of the PHP parser (used for reads) and the output of Parsoid (used for edits in VE, Content Translation) more consistent since Parsoid already uses HTML5 semantics. One of our goals is to make the two outputs fully consistent with each other and use one parser for both reads and edits.

-- Parsing>Parsing</>|The Parsing Team

Volunteers available to support this effort
''Community Liaisons invite interested Wikimedians to please add their name in the sections below and support their community engagement efforts. Thank you. As with tforce>Help:VisualEditor/Community Taskforce</>|similar past initiatives, signing up is optional.''
 * 1) I am available to test with the ParserMigration tool.
 * 2) (add your signature here)
 * 3) I am available to fix templates.
 * 4) Jonesey95 (talk) 16:10, 14 November 2016 (UTC)
 * 5) Samuele2002 (talk)
 * 6) * . Please see https://phabricator.wikimedia.org/T161341 for templates that would need fixing. The linter category mentioned in that ticket should start populating later this week. SSastry (WMF) (talk) 18:09, 12 April 2017 (UTC)
 * 7) **Unable to find a link to or mention of any such category. As with most phab tickets, the insider lingo is impenetrable. Jonesey95 (talk) 04:58, 13 April 2017 (UTC)
 * 8) **The link will show up later today. But, is the explanation here any better? But, based on your feedback, we'll try to improve the explanation. See this example on testwiki (and examine the source). You will see that there is no red border around the table. SSastry (WMF) (talk) 12:40, 13 April 2017 (UTC)
 * 9) ***Jonesey95, did that help? Elitre (WMF) (talk) 15:30, 27 April 2017 (UTC)
 * 10) ***Also, we had to disable Linter on large wikis temporarily because of performance issues. It will come back in 2 weeks. Also, the Linter help pages have more information now. SSastry (WMF) (talk) 15:33, 27 April 2017 (UTC)
 * 11) ****Thanks for the new Linter pages. They are as well written as documentation pages tend to get around here, and they should serve as good technical conversation-starters in en.WP gnome hangouts. I am looking forward to the arrival of these features on en.WP. As I mentioned on a page recently (I can't recall where), I believe that https://phabricator.wikimedia.org/T157670 is still blocking progress on creating full lists of pages with these errors. Pages that haven't been null-edited in a while may not show up on the lists. Jonesey95 (talk) 18:08, 27 April 2017 (UTC)
 * 12) ***** https://phabricator.wikimedia.org/T161556 will address this SSastry (WMF) (talk) 18:16, 27 April 2017 (UTC)
 * 13) ******Interesting. It appears that we have four versions of essentially the same feature request. I hope someone can implement at least one of the recommended fixes. Jonesey95 (talk) 04:06, 28 April 2017 (UTC)
 * 14) I am available to study and discuss fixes to templates.
 * 15) Jonesey95 (talk) 16:10, 14 November 2016 (UTC)
 * 16) I am available to spread the word among my community.
 * 1) Jonesey95 (talk) 16:10, 14 November 2016 (UTC)
 * 2) I am available to spread the word among my community.