User:Cmcmahon(WMF)

QR notes

Done:

Testing
 * Browser tests managed in extension repos
 * VisualEditor, MobileFrontend, Wikidata, CirrusSearch, Flow, ULS, TranslateWiki, MultimediaViewer, ?
 * Few remaining generic tests to be moved to core at some point.

Test environment
 * Beta labs used by every project. Parsoid was the last
 * Most bugs exposed in beta labs
 * Test2/mediawiki.org still useful for staging. Example: VE bug introduced on Thursday was stopped before prod because of test2wiki.

Expanding automated tests beyond browser:


 * Currently monitoring upload API in production Commons and beta Commons
 * Using API to create test data articles in target wiki at run time
 * Running browser tests headless with Xvfb on Linux with Firefox and Chrome

Current pain points:
 * Sharing common code among repos using /mediawiki/selenium
 * Migration underway
 * Targeting bare wiki, local wiki, development environment for testing
 * Requested by Mobile team, Mediawiki release team, WMF CI, etc.
 * Cloudbees Jenkins creating too many false failures and no way to diagnose the issues
 * Slowly making it possible to migrate to WMF Jenkins, see https://www.mediawiki.org/wiki/Browser_testing/architecture

Misconceptions:
 * Beta labs is seen as unreliable; in fact, beta falls over because we find real bugs there
 * Browser tests are too successful. Teams not writing unit tests, not writing integration tests, not doing monitoring, because browser tests are seen as a universal panacea.