Parsoid/Round-trip testing

The Parsoid code includes a round-trip testing system that tests code changes, composed of a server that gives out tasks and presents results and clients that do the testing and report back to the server. The code is in Parsoid/tests/server and Parsoid/tests/client.

There's a publicly accessible instance of the server at http://parsoid-tests.wikimedia.org/. It currently tests a representative (~160000) set of pages from different wikipedia languages.

Private setup
The instructions to set up a private instance of the round-trip test server are in Parsoid/tests/README. A MySQL database is needed to keep the set of pages and the testing results.

RT-testing setup
Coordinator runs on ruthenium. RT-testing clients run on ruthenium and commit suicide when the revision of that checkout changes. You need access to bastion on wikimedia.org to access ruthenium.

The clients are managed/restarted by upstart. Config in /etc/init/parsoid-rt-client.conf.

To {stop,restart,start} all clients on a VM (not normally needed): sudo service parsoid-rt-client stop sudo service parsoid-rt-client restart sudo service parsoid-rt-client start

Client logs are in /var/log/upstart/parsoid-rt-client.log.

Updating the code to test (and being run by the clients)
Normally, this is not needed since the code is updated every midnight (PST). But, if you need to update rt-testing code at any other time, run the following on ruthenium: sudo -u parsoid-rt /home/parsoid-rt/update-code.sh

Clients commit suicide when they notice that the code has changed, and upstart restarts them based on the new code. To restart them manually on a client node:

sudo service parsoid-rt-client restart RT test server logs are in /var/log/testreduce/testreduce.log.

Updating the round-trip server code
The rt-server code runs off a clone of the parsoid repo since our rt-server code is still part of the core parsoid repo. /usr/lib/testreduce is a symlink into /usr/lib/parsoid-clone/tests/server cd /usr/lib/parsoid-clone git pull sudo service testreduce restart

Todo / Roadmap
Please look at the general Parsoid roadmap.

Server UI and other usability improvements
We recently changed the server to use a templating system to separate the code from the presentation. Now other improvements could be done on the presentation itself.

Ideas for improvement:

 * Improve pairwise regressions/fixes interface on commits list . Done!
 * Flag certain types of regressions that we currently search for by eye: create views with
 * Regressions introducing exactly one semantic/syntactic diff into a perfect page, and
 * Other introductions of semantic diffs to pages that previously had only syntactic diffs.
 * Improve diffing in results views:
 * Investigate other diffing libraries for speed,
 * Apply word based diffs on diffed lines,
 * Diff results pages between revisions to detect new semantic/syntactic errors,
 * Currently new diff content appears before old, which is confusing; change this.

Some dsh tricks
Restart all rt servers dsh -cf /home/gwicke/rtclients sudo service parsoid-rt-client restart

Show memory usage of all node processes on all nodes dsh -M -cf /home/gwicke/rtclients ps -C nodejs -o rss= | sort -n -k2