Wikimedia Release Engineering Team/status

Last update on: 2014-07-monthly

2013-12-03

The Wikimedia Release and QA team had their quarterly review in November. You can see the work items completed (and started) along with plans for next quarter on the quarterly review page.

2013-12-monthly

In December, the latest and greatest version of MediaWiki was released, 1.22. This was lead by Mark Hershberger and Markus Glaser, working as the MediaWiki release team, along with help from the Wikimedia Foundation Release and QA team (specifically Greg Grossmeier and Antoine Musso). Of course, this was only possible because of the great work by all of the MediaWiki developers.

The QA team, along with Multimedia team, is working on API level tests starting with UploadWizard. This is close to being done. Another API level testing activity is Parsoid, with help from VE and CI (Antoine) teams.

You can take a look at the first draft of the updated Development and Deployment process flow chart.

2014-02-monthly

Wikimedia development and deployment flowchart

The Release and QA team had their latest quarterly review on February 13. Highlights from the meeting include:

• We will be hiring two new positions (a QA Automation Engineer and a Test Infrastructure Engineer).
• We will process through all pain points from the Development and Deployment process review.
• We will continue performing incremental improvements to the current deployment script (known as "scap") to better inform future deployment tooling work.
• We will create a way for tests to create fake/stub data (for use in throw-away/one-off test instances).
• We will make it so our browser tests are more accurate cross testing and production environments.

Notable progress on things with visuals includes an updated Development and deployment flowchart (opposite), as well as an auto-generated version.

2014-03-monthly

The Beta Cluster has been migrated from the Tampa data center to the Ashburn data center. In the move, a ton of cleanup and Puppetization work was done. This will make future Beta Cluster work easier. In addition, the Beta Cluster is getting closer to a place where we can test our current main deployment tool known as "scap" along with future/other deployment tools.

The team continued on the rewrite of scap into python (from Bash scripts + PHP), improving both performance and maintainability in addition to being in a better position to move to a new tool in the future. We have also started doing SWAT deploy windows twice a day (Monday to Thursday) which has greatly increased momentum for many developers who would otherwise have to wait until the weekly deployment cycle.

2014-04-monthly

Presentation slides of the Release engineering and QA quarterly review. See also the agenda and minutes.

The tool that deploys code in production ("scap") is now used to deploy/update code on the Beta cluster, removing another difference between the Beta cluster and production along with providing us an environment to safely test changes to our deployment system(s). We converted more scap code to python (scap-rebuild-cdbs and mw-update-l10n), and moved a ton of Jenkins jobs from the Cloudbees Jenkins to our self-hosted instance; we're on target to end the use of Cloudbees' Jenkins in the next two weeks. We also made significant progress on the two open positions (Release Engineer and QA Automation).

2014-05-monthly

The Release and QA team expanded this past month with the addition of Mukunda Modell as a new Release Engineer. His first work is addressing the remaining technical issues blocking our adoption of Phabricator as a WMF-wide tool. Antoine Musso has begun drafting an RFC to outline how the WMF would support isolated test environments for automatic builds.

The WMF kicked off the second RFP for the release management of MediaWiki. Chris McMahon fleshed out tooling for creating test data at test time more widely along with building off of Antoine's work for using the appropriate version of browser tests for testing specific mediawiki test installs (browser tests are versioned along with the other code).

2014-06-monthly

The Release and QA Team had their mid-quarter check-in on June 27. Phabricator work is progressing nicely. The latest MediaWiki tarball release (1.23) was made and the second RFP started and is close to completion. We are moving to only WMF-hosted Jenkins for all jobs, and we are working with the MediaWiki Core and the Operations teams on HHVM-related integration (both for deployment and for the Beta Cluster).

2014-07-monthly

This month, the Release and QA Team became the Release Engineering Team, mostly reflecting the transition of this team from being made up of members of other distinct teams to that of a coherent self-contained (mostly) team. This will, hopefully, allow better coordination of "Release" and "QA" things (broadly spreaking).

A lot of progress was made on making Phabricator suitable as a task/bug tracking system for Wikimedia projects. You can see the work to be sorted and completed at this workboard.

The Beta Cluster now runs with HHVM, bringing us much closer to full HHVM deployment. In addition, the Language Team deployed the new Content translation system on the Beta Cluster with the help of the Release Engineering team.

The second round of public RFP for third-party MediaWiki release management was conducted and concluded.

We now no longer use the third-party Cloudbees service for any of our Jenkins jobs and run all jobs locally. This will enable us to better diagnose issues with our build process, especially as it pertains to our browser tests (which still mostly run on SauceLabs).