Quality Assurance/Status 2012-2015

2012

2012-10-19

Zeljko Filipin has an initial set of browser tests checked in to Github. These will be moved into Gerrit soon.

2012-10-monthly

QA Engineer Željko Filipin has made great improvements to the existing automated browser tests and has created some new Mobile tests as well. Mobile QA Engineer Michelle Grover is creating an automatable regression test suite for MobileFrontend. These tests are currently running as builds under a hosted instance of Jenkins, with the intention of moving them to the WMF Continuous Integration environment pending upgrades to the machines hosting gerrit and Jenkins.

2012-11-06

Filipin and McMahon now have browser tests in Gerrit, in qa/browsertests . They are porting existing tests from RSpec to Cucumber. The UploadWizard test is the most complex and they're in the midst of porting it. See Qa/test backlog.

2012-11-13

Ported the toughest test (UploadWizard) to Cucumber, created backlog of tests to be automated, will introduce Cucumber to feature development soon. Chris, this month, aims to socialize Cucumber acceptance test-writing (using "given-when-then" format) in browser testing, to get developers to more regularly write these tests as they work and commit. He'll be talking to Fabrice, Arthur, Siebrand, James Forrester, and probably Tomasz. Coverage goals: extensions that are on beta or test2 right now.

2012-11-27

Working on architecture, creating basic new tests, refactoring those for cleanliness. Public announcement will be coming very soon, when the existing exemplar tests are in final or near-final form.

2012-11-30

Noisy tests failing for known reasons have been removed from the suite, and the suite is now completely green. Adding more tests is immanent. Browser tests identified a serious regression in UploadWizard running on test2 and prevented its release to production.

2012-11-monthly

In November, the QA team created a backlog of tests to be automated, ported existing tests from RSpec to Cucumber, and is now working on browser testing architecture, creating basic new tests (see the qa/browsertests repository in Gerrit), and refactoring tests for cleanliness. Chris McMahon began discussing automated browser tests with Wikimedia tech managers to get developers writing those tests as they develop extensions deployed on Wikimedia sites; public announcement will be coming very soon, when the existing example tests are in final or near-final form. Noisy tests failing for known reasons have been removed from the suite, which is now completely green (that is, passing); the team will soon be writing and adding more tests. Browser tests in November identified a serious regression in UploadWizard running on test2 and prevented its release to production.

2012-12-10

Refactoring Jenkins builds for granularity, fixing last bits of tests known to be broken. Starting to expand test coverage, possible candidates PageTriage, Echo, VE, etc.

2012-12-18

Refactoring Jenkins builds for granularity, fixing last bits of tests known to be broken. Starting to expand test coverage, possible candidates PageTriage, Echo, VE, etc.

2012-12-28

Quim and Chris have a goal to get all the browser automation docs refactored by January 4th.

2012-12-monthly

After its announcement about the state of automated browser testing on wikitech-l, the QA team continued to expand test coverage, improve system and project documentation, and publicize and socialize the project by means of the "Browser Testing" MediaWiki Group.

2013

2013-01-22

Architecture and configuration are stable, focus now is on increasing test coverage by making existing tests more extensive and by covering new features.

2013-01-29

Added browser test coverage for the Math extension. (Math was briefly broken after the EQIAD migration, this is a valuable regression test). Adding more coverage all the time, we've instituted a weekly pair-programming session every Friday morning PST.

2013-01-monthly

Architecture and configuration for browser testing are now stable, and the focus shifted to increasing test coverage by making existing tests more extensive and covering new features. An example is the Math extension, which was briefly broken after the data center migration. The team has also instituted a weekly pair-programming session every Friday.

2013-02-05

Automated test for Math extension exposed an issue with the production deployment of 4 February. Proposing adding support for PDBHandler, a labs projects in development. Several new tests in the pipeline, one in conjunction with a new volunteer.

2013-02-12

Will be cleaning up js error check causing failures in IE8 and below. Will be adding new tests also. Timeouts in test environments have been causing extra failures recently

2013-02-25

Created several tests in support of Language team with more to come, including first test for VisualEditor. Browser test tech talk in SF Thu Feb 28

2013-03-05

Added E3 tests. Preparing for test event to increase the backlog. Sophisticated tests for Language need tweaking/research.

2013-02-monthly

Added E3 tests. Preparing for test event to increase the backlog. Sophisticated tests for Language need tweaking/research.

2013-03-12

Exercise to build the test backlog using Search as an example was successful, test backlog for search now contains wide variety of regression tests for automation. Browser tests continue to identify significant regression issues with new features in particular.  

2013-03-monthly

We held an exercise to build the test backlog using Search as an example which did build the test backlog for search; it now contains wide variety of regression tests for automation. But the exercise didn't attract as many volunteers as we'd like; we're looking for ways to address that in future exercises. Browser tests continue to identify significant regression issues, with new features in particular. We are extending browser test coverage to move-page functions, PDFs, and Search with a volunteer from the March exercise.

2013-04-02

Move-test page is nearly complete. Investigating issues with Selenium IE driver. As noted above, pointing tests to beta as well as test2wiki.

2013-04-08

High priority is to target beta cluster as well as test2wiki. Turned off specific tests failing in IE6/7/8 until we can fix them reliably.

2013-04-16

Making significant changes to browser tests to run and pass on both test2wiki and beta cluster.

2013-04-monthly

We created a number of new builds to point browser tests to the beta cluster as well as test2wiki. We also normalized user strings for test purposes on test2wiki and beta cluster wikis. We added new tests for the Preferences/Appearance tab and SUL login, and a volunteer contributor added a test for PDF manipulation.

2013-05-21

Automated tests continue to find issues, although those issues prevent many builds from being green during maintenance and fixing. Preparing for OPW work to automate tests for VisualEditor, and some ULS in progress also.

2013-05-monthly

In May, we added beta labs as a target for automated browser tests, which allowed us to create tests for the Universal Language Selector at the Hackathon in Amsterdam. We shored up a lot of Jenkins builds at the hackathon as well. We created our first test for VisualEditor, and we are looking forward to working with Rachel Thomas (as part of the Outreach Program for Women) to create more.

This month QA added new browser tests for UniveralLanguageSelector contributed by the WMF Language team and tests for Mobile from that team also, as well as browser test contributions from volunteers. We created new builds in Jenkins to run browser tests against IE10. We created tests for Visual Editor, including some from our intern with the Outreach Program for Women.

2013-06-monthly

This month, the QA team added new browser tests for UniveralLanguageSelector and for Mobile (contributed by the Language engineering and Mobile engineering teams, respectively), as well as browser test contributions from volunteers. We created new builds in Jenkins to run browser tests against IE10. We created tests for VisualEditor, including some with our intern with the Outreach Program for Women.

Beta labs now supports more tests than test2wiki, including VisualEditor, MobileFrontend, UniversalLanguageSelector, ArticleFeedbackv5, Wikilove, and more. We made our builds green in Jenkins by researching and codifying in the tests our various levels of support for various versions of Internet Explorer. We merged our first contributions from Amir Aharoni in the Language team and Tomislav Plavcic from the QA mail list. The automated browser test suite continues to identify important issues in the software.

2013-07-monthly

In July we added coverage for a number of features, including VisualEditor, UniversalLanguageSelector, and Mobile Search. We are making extensive use of beta labs as well as the test2wiki test environment. Our automated browser tests continue to identify important issues during feature development.

This month QA added new browser tests for UniveralLanguageSelector contributed by the WMF Language team and tests for Mobile from that team also, as well as browser test contributions from volunteers. We created new builds in Jenkins to run browser tests against IE10. We created tests for Visual Editor, including some from our intern with the Outreach Program for Women.

2013-08-monthly

This month saw a significant change to the structure and organization of browser tests, with tests and builds for CirrusSearch, UniversalLanguageSelector, and VisualEditor following the example of MobileFrontend and now residing in the git repositories for those extensions, rather than in the /qa/browsertests repository. This creates opportunities for more frequent and more accurate Jenkins builds of the tests, while also reducing the overhead required for analyzing test failures.

2013-09-monthly

This month saw significant improvements to both coverage and speed in our tests for VisualEditor. We are collaborating with the Language team on browser tests for the UniversalLanguageSelector extension and Translatewiki.net. We created our first tests for the new Flow feature and are in the process of supporting Flow fully in a reference test environment. We presented yet another of our ongoing series of training sessions, this one live in San Francisco.

2013-10-monthly

October saw a major change to the architecture of the browser tests. Since they are now in many different repositories (VisualEditor, UniversalLanguageSelector, MobileFrontend, Flow, etc.), we consolidated the code shared among the various tests in all the repositories into a single library. This allows all the tests, present and future, to use the best and most up-to-date infrastructure. As for the tests themselves, we expanded coverage and identified issues for VisualEditor, Flow, gadgets, Language features, MobileFrontend, and more.

2013-11-monthly

In November, we added significant browser test coverage for the Flow project, and the addition of Jeff Hall to WMF staff brought a focus to testing VisualEditor. Browser tests now reside in ten different repositories across WMF projects. November saw a increased browser test coverage for the Language, VisualEditor, and Flow projects, among others. The diversity of browser tests in project repositories has been a force behind great improvements in infrastructure, with code shared among the projects now residing in the repository at mediawiki/selenium.

2013-12-monthly

Besides ongoing regression testing of Wikipedia features in cross-browser tests, in December we made the first steps for new abilities like testing geolocation for Mobile tests, testing and monitoring upload ability in production, adding the ability to create test data via the API, running tests in PhantomJS on the WMF Jenkins server, and monitoring the Beta labs test environment for fatal errors.

2014

2014-01-monthly

In January, we had a number of contributions from the students of Google Code-in, from tests to Jenkins configuration to documentation. We released two entirely new features: one test that monitors the file upload API interface on both production Commons and beta labs Commons, and another test that monitors fatal errors in Beta Labs. We are very close to announcing general availability for two other new features: the ability to run tests headless using Firefox under Xvfb, and the ability to create test data like wiki pages in the target wiki at run time.

2014-02-monthly

Our test coverage of MediaWiki extensions continues to prove itself. In February, using the automated browser tests running against beta labs and test2wiki, we found and fixed several critical errors that would have disrupted production wikis severely if they had been released.

2014-03-monthly

Besides a particular focus on MobileFrontend browser tests in March, we have also made available some new features, in particular shared code to upload files properly in all browsers, the ability to check for ResourceLoader problems in any test in any repository, and a basic wrapper in order to use the Mediawiki API from within browser tests to set up and tear down test data.

2014-04-monthly

The QA team released a number of new browser test features, including the ability to create test data in the target wiki at runtime. This feature was immediately put into use by the MobileFrontend team in their browser test suite. A complete list of shared features available to any browser tests in any extension repository is available.

2014-05-monthly

Now that we have the ability to address the MediaWiki API from the browser testing framework, we have changed the existing test suites to use this powerful tool. This not only gives us a test of the API itself, but also makes the browser tests faster and more reliable. Furthermore, it allows us to easily create a set of acceptance tests that will pass on any MediaWiki installation regardless of what extensions exist or what language the wiki is. We plan for these acceptance tests to eventually become part of MediaWiki core, and our existing tests continue to expose important issues in Wikimedia software development projects.

2014-06-monthly

After two years of using a third-party host to run browser test builds in Jenkins, this month we have completed the migration of those builds to Jenkins hosted by the Wikimedia Foundation. Hosting our browser test builds ourselves gives us more control over every aspect of running the browser tests, as well as the potential to run them faster than previously possible. Particular thanks to Antoine Musso, whose work made it possible. Simultaneously, we have also ported all of the remaining tests from the /qa/browsertest repository either to /mediawiki/core or to their relevant extension. This gives us the ability to package browser-based acceptance tests with the release of MediaWiki itself. After more than two years evolving the browser testing framework across WMF, the /qa/browsertests repository is retired, and all if its functions now reside in the repositories of the features being tested.

2014-07-monthly

The browser tests are now all integrated with builds on the Wikimedia Jenkins host. We added browser tests for MediaWiki core that will validate the correctness of a MediaWiki installation regardless of language, or of what extensions may or may not exist on the wiki, so that the tests may be packaged with the distribution of MediaWiki itself and used on arbitrary wikis. We saw a lot of browser test activity for Flow development, and we are preparing to support even more extensions and features in the very near future.

2014-08-monthly

This month, we continued to build out and adjust the new browser test builds on Jenkins. We saw updates to tests and issues identified for UploadWizard, VisualEditor, Echo, and MobileFrontend. New tests for GettingStarted pointed out a need to update our Redis storage on the beta cluster. We are currently monitoring an upstream problem with Selenium/Webdriver and IE11 on behalf of VisualEditor, as VE support for IE11 is coming soon.

2014-10-monthly

This month we began a significant project for improving and refactoring existing browser tests. Starting with the Echo and Flow repositories, we intend to update and improve the browser tests across all the repos to be more modern, flexible, maintainable, and reliable. Our new Ruby style checker Rubocop in all of the Jenkins builds will help greatly in this effort, as will major improvements to the architectures of the features being tested, of which VisualEditor is a notable example.

2014-11-monthly

In November the CentralNotice repo acquired its first browser tests. In addition to adding new test coverage, we continue to refactor existing tests across all of our repositories. While the primary purpose of this refactoring is to update all of the tests' assertions to RSpec3 syntax from RSpec2, we are also taking the time to address technical debt and sort other issues in the tests such as removing inefficient code and replacing explicit wait statements with dynamic wait-for statements. This not only improves the speed at which the tests run, but also helps immensely with maintenance and usability into the future.

2015

2015-03-monthly

Quality Assurance/Browser testing/status