Parsoid/Round-trip testing

From MediaWiki.org
Jump to: navigation, search

The Parsoid code includes a round-trip testing system that tests code changes, composed of a server that gives out tasks and presents results and clients that do the testing and report back to the server. The code is in the testreduce repo which is an automatic mirror of the repo in gerrit. The roundtrip testing code on ruthenium has been fully puppetized.

There's a publicly accessible instance of the server at http://parsoid-rt-tests.wikimedia.org/. It currently tests a representative (~160000) set of pages from different wikipedia languages.

Private setup[edit]

The instructions to set up a private instance of the round-trip test server can be found here. A MySQL database is needed to keep the set of pages and the testing results.

RT-testing setup[edit]

Coordinator runs on ruthenium. RT-testing clients run on ruthenium and commit suicide when the revision of that checkout changes. You need access to bastion on wikimedia.org to access ruthenium.

The clients are managed/restarted by systemd and the config is in /lib/systemd/system/parsoid-rt-client.service. Please do not modify the config on ruthenium directly (they will be overwritten by puppet runs every 30 minutes). Any necessary changes should be made in puppet and deployed.

To {stop,restart,start} all clients on a VM (not normally needed):

sudo service parsoid-rt-client stop
sudo service parsoid-rt-client restart
sudo service parsoid-rt-client start

Client logs are in systemd journals and can be accessed as:

### Logs for the parsoid-rt-client service
# equivalent to tail -f <log-file>
sudo journalctl -f -u parsoid-rt-client
# equivalent to tail -n 1000
sudo journalctl -n 1000 -u parsoid-rt-client

### Logs of the parsoid-rt testreduce server
sudo journalctl -f -u parsoid-rt

### Logs for the parsoid service
sudo journalctl -f -u parsoid

In the current setup, the testreduce clients talk to a global parsoid service that runs on ruthenium. So, look at the logs of the parsoid service to find problems / bugs with the parsoid code being tested. These logs are also mirrored to Kibana which you can find on this dashboard.

Updating the code to test (and being run by the clients)[edit]

To update rt-testing code, run the following on ruthenium:

update_parsoid.sh

This updates the parsoid checkout, restarts the parsoid service, and the parsoid-rt-client service.

Updating the round-trip server code[edit]

The rt-server code lives in /srv/testreduce and runs off the ruthenium branch.

After review/merge on master, checkout the ruthenium branch locally. Merge master into ruthenium. If you need to update node_modules/ do that as well, and commit it, then push the branch to gerrit (you should have push rights).

cd /srv/testreduce
## Please verify you are in the ruthenium branch before the git pull
git pull
sudo service parsoid-rt restart

Todo / Roadmap[edit]

Please look at the general Parsoid roadmap.

Server UI and other usability improvements[edit]

We recently changed the server to use a templating system to separate the code from the presentation. Now other improvements could be done on the presentation itself.

Ideas for improvement:[edit]
  • Improve pairwise regressions/fixes interface on commits list bug 52407. Done!
  • Flag certain types of regressions that we currently search for by eye: create views with
    • Regressions introducing exactly one semantic/syntactic diff into a perfect page, and
    • Other introductions of semantic diffs to pages that previously had only syntactic diffs.
  • Improve diffing in results views:
    • Investigate other diffing libraries for speed,
    • Apply word based diffs on diffed lines,
    • Diff results pages between revisions to detect new semantic/syntactic errors,
    • Currently new diff content appears before old, which is confusing; change this.