User:Cmcmahon(WMF)

QR notes Feb 2014

Done:

Testing
 * Browser tests managed in extension repos
 * VisualEditor, MobileFrontend, Wikidata, CirrusSearch, Flow, ULS, TranslateWiki, MultimediaViewer, ?
 * A few remaining generic tests to be moved out to core etc. at some point.

Test environment
 * Beta labs used by every project. Parsoid was the last not to use beta.
 * Most bugs exposed in beta labs
 * test2/mediawiki.org still useful for staging. Example: VE bug introduced on Thursday was stopped before prod because of test2wiki.

Expanding automated tests beyond browser:


 * Currently monitoring upload API in production Commons and beta Commons
 * Using API to create test data articles in target wiki at run time
 * Running browser tests headless with Xvfb on Linux with Firefox and Chrome

Current pain points:
 * Sharing common code among repos using /mediawiki/selenium
 * Migration underway
 * Targeting bare wiki, local wiki, development environment for testing
 * Requested by Mobile team, Mediawiki release team, WMF CI, etc.
 * Cloudbees Jenkins creating too many false failures. Diagnosis is cumbersome, fixes have to go through Cloudbees support. Chrome in particular causes problems, as does Net::HTTP::Persistent
 * Slowly making it possible to migrate to WMF Jenkins, see https://www.mediawiki.org/wiki/Browser_testing/architecture
 * test2wiki/mw.o lagging beta labs updates
 * Investigating pulling master branch for beta labs vs. release branch for test2wiki

Messages:
 * Beta labs is seen as unreliable, "brittle and moody" from one recent critic; in fact, beta falls over mostly because we release real bugs there
 * But still needs further work to emulate production, for example db slave arrangements
 * Updating beta could still be more reliable
 * Are browser tests too successful? Dev teams are not writing unit tests, not writing integration tests, not doing monitoring, because browser tests are seen as the only automated tests.  See http://martinfowler.com/bliki/TestPyramid.html