Parsing/Replacing Tidy/Linter/Stats/Pixel Diff Testing Stats

From mediawiki.org

We are stopping these diffing runs as of May 29th.

Every week, we run a mass pixel-diff testing on about 72K pages from 60 different wikis from 4 different projects. We generate a rendering of the (latest revision of the) test page that uses Tidy and another rendering of the test page that uses Remex and then compare the two images via the uprightdiff package. If the two images are not completely identical, the diffing algorithm checks if the differences can be accounted for by purely vertical whitespace shifts.

The table below shows how the results changed since August 2017 when we first started these tests. At that time, our internal target for complete Tidy replacement was 95% pages render identically (after accounting for vertical whitespace shifts). Because of shortcomings of our tooling, it is not possible to get to 100% because of false positives and noisiness in tests.

As the table shows, as editors fix linter-identified issues on pages, this percentage has been slowly increasing.

Results of mass pixel-diff testing run, on average, once every week
Date % vertical

whitespace

shifts only

% pixel-perfect

rendering

Notes
2017-08-03 91.90 63.23
2017-08-07 91.94 62.89
2017-08-10 91.77 63.52
2017-08-16 91.82 62.96
2017-08-21 91.87 62.98
2017-08-30 91.77 63.07
2017-09-04 91.81 63.07
2017-09-11 91.78 62.99
2017-09-18 91.72 62.95
2017-09-25 91.94 62.84
2017-10-02 92.11 62.97
2017-10-09 92.27 62.73
2017-10-16 92.35 63.23
2017-10-23 92.50 63.50
2017-10-30 92.23 63.84
2017-11-02 93.53 67.73 This big bump is because of fixes to the visual diff testing setup.

Redirects weren't being following for both Tidy and Remex and was causing false diffs to be reported.

2017-11-06 93.56 67.71
2017-11-13 93.68 67.87
2017-11-20 93.72 68.00
2017-11-27 93.91 68.05
2017-12-04 94.39 68.06
2017-12-11 94.47 68.10
2017-12-18 94.53 68.25
2017-12-25 94.62 68.16
2018-01-01 94.61 68.32
2018-01-08 94.64 68.27
2018-01-15 94.65 68.27
2018-01-22 94.68 68.30
2018-01-29 94.74 68.32
2018-02-05 94.83 68.42
2018-02-12 94.89 68.33
2018-02-19 94.87 68.32
2018-02-26 94.91 68.44
2018-03-05 94.94 68.46
2018-03-12 94.91 68.43
2018-03-19 94.92 68.43
2018-03-26 95.59 70.82 This is after the whitespace trimming patch was deployed. Surprised to see the big change here.
2018-04-02 95.64 70.95
2018-04-09 95.63 70.94
2018-04-16 95.63 71.00
2018-04-23 95.66 70.90
2018-04-30 95.66 70.93
2018-05-07 95.72 70.78
2018-05-14 95.66 70.68
2018-05-21 95.63 70.75
2018-05-29 95.68 70.86