Quality Assurance/Status 2012-2015

Last update on: 2012-12-10

2012-10-19
Zeljko Filipin has an initial set of browser tests checked in to Github. These will be moved into Gerrit soon.

2012-10-monthly
QA Engineer Željko Filipin has made great improvements to the existing automated browser tests and has created some new Mobile tests as well. Mobile QA Engineer Michelle Grover is creating an automatable regression test suite for MobileFrontend. These tests are currently running as builds under a hosted instance of Jenkins, with the intention of moving them to the WMF Continuous Integration environment pending upgrades to the machines hosting gerrit and Jenkins.

2012-11-06
Filipin and McMahon now have browser tests in Gerrit, in qa/browsertests. They are porting existing tests from RSpec to Cucumber. The UploadWizard test is the most complex and they're in the midst of porting it. See Qa/test backlog.

2012-11-13
Ported the toughest test (UploadWizard) to Cucumber, created backlog of tests to be automated, will introduce Cucumber to feature development soon.

Chris, this month, aims to socialize Cucumber acceptance test-writing (using "given-when-then" format) in browser testing, to get developers to more regularly write these tests as they work and commit. He'll be talking to Fabrice, Arthur, Siebrand, James Forrester, and probably Tomasz. Coverage goals: extensions that are on beta or test2 right now.

2012-11-27
Working on architecture, creating basic new tests, refactoring those for cleanliness. Public announcement will be coming very soon, when the existing exemplar tests are in final or near-final form. 

2012-11-30
Noisy tests failing for known reasons have been removed from the suite, and the suite is now completely green. Adding more tests is immanent. Browser tests identified a serious regression in UploadWizard running on test2 and prevented its release to production.

2012-11-monthly
In November, the QA team created a backlog of tests to be automated, ported existing tests from RSpec to Cucumber, and is now working on browser testing architecture, creating basic new tests (see the  repository in Gerrit), and refactoring tests for cleanliness. Chris McMahon began discussing automated browser tests with Wikimedia tech managers to get developers writing those tests as they develop extensions deployed on Wikimedia sites; public announcement will be coming very soon, when the existing example tests are in final or near-final form. Noisy tests failing for known reasons have been removed from the suite, which is now completely green (that is, passing); the team will soon be writing and adding more tests. Browser tests in November identified a serious regression in UploadWizard running on test2 and prevented its release to production.

2012-12-10
Chris McMahon summarized the status of browser test automation in an email to wikitech-l.