Wikimedia Release Engineering Team/Process
This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date.
Who we are
We get software from the developer to production as quickly, easily, and safely as possible.
We are big fans of the values of the Agile Manifesto:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
Software needs to be in front of users for it to have value. The longer code spends in a development environment, a test environment, a deployment queue, the more it costs and the less value it has. Release Engineering is in the business of making that process as fast as it can reasonably be made.
People creating software for users should have all the tools necessary right to hand, without having to struggle with tasks and chores unrelated to development. From planning tools and shared development environments and code repositories to test environments to deployment processes, Release Engineering works to make the steps from idea to users as easy and as automatic as they can be made.
Speed and ease are important, but not at the cost of quality. Release Engineering includes testing processes of all kinds, from code linters to continuous integration to automated system tests to production scripting. Furthermore, we take Quality Assurance seriously, in the sense that QA has to do with improvements to methodology and process throughout the development cycle.
|Early in the dev process --->||---> After merge to master branch --->||---> Production|
|Planning & Tracking||Shared Development Environment||Continuous Integration||Beta Cluster||Browser Test Automation||Browser Test Automation||Exploratory Testing||Exploratory Testing||Deploying to Production||Management|
|Person||Mukunda Modell||Dan Duvall||Antoine||Antoine||Željko||Chris||Elena||Rummana||Chad||Greg|
|Tools and systems||Phabricator||Vagrant, Puppet||Jenkins, Zuul||Puppet, sysadmin||page-object, Jenkins job builder||page-object, Jenkins, Sauce Labs, beta cluster||beta cluster, test2wiki||Gerrit, Phabricator, beta cluster, test2wiki and enwiki||Gerrit, scap||SWAT, logs, schedules|
|Activities||Maintenance, new features in response to dev teams' requests.||Maintenance, improvements to code and infrastructure.||Maintenance, new features||Maintenance, updates to improve performance and emulation of prod systems||Jenkins infrastructure, shared code, community training||Failure analysis, test design, internal training, process||Exploratory testing, reporting issues, bug triage||Exploratory testing, reporting issues, bug triage,bug verification.||Weekly deployments, SWAT when necessary||Oversee production deployments, do schedules, planning|
|Collaborates with||Ops, ECT||Core, Features teams, browser test||Core, Ops, Labs, Devs, Volunteers||Ops, Labs||Language, Search, Wikidata, ECT||Mobile, VE, Flow, MMV, Labs, Team practices||Editing||Editing and Mobile Apps team||Core, Ops||Everyone|
|Working on||Expand Phab capabilities||Vagrant in CI, more shared code||Performance, browser tests voting, CI infra||Better prod emulation, staging||Browser tests voting||
||ET for other WMF teams; help with automated tests||ET for other WMF teams,making browser tests complete and reliable for high-priority features in VE for Q3 ,helping with bug triage||More automation||More automation, more information|
Tools and Frameworks
Release Engineering oversees many of the tools and frameworks that help in the delivery of software features, from development environments and test frameworks to deployment scripts. Creating and maintaining reliable automation is a constant concern for us on every front.
Here is a list of tools and frameworks that people in Release Engineering understand and maintain. We may not be the only ones who know about each item, but we are all knowledgeable to a certain extent about all of the systems listed here.
|Initial||Tools and frameworks||People with answers|
|B||Browser tests||Chris, Željko, Rummana|
|BC||Beta cluster||Chad, Mukunda, Tyler, Chris, Antoine|
|ET||Exploratory Testing||Rummana, Elena (Željko, Chris also)|
|J||Jenkins/Zuul||Antoine, Željko, Tyler|
|SC||SCAP||Greg, Mukunda, Chad|
Teams and Tools
Release Engineering has direct, reciprocal relationships with at least ten other teams within WMF. Here is a table that attempts to capture the nature of those relationships in terms of the tools and frameworks that we use in common.
|Editing||B BC ET J PH SC SL SW V|
|Mobile Web/Apps||B BC ET J PH SC SL SW V|
|Services/Parsoid||BC J PH SW|
|Collaboration||B BC J PH SC SL SW V|
|Language||B BC J PH SC SL V|
|Core||BC J P PH SC SW|
|Multimedia||B BC J PH SC SL SW V|
|Engineering Community Team||B(?) BC PH|
|Team Practices Group||PH|
|Ops||BC P SC|
How Release Engineering Works
Release Engineering creates, manages, and supports the tools and practices that allow developers to create software easily, manage software safely, and get software to users quickly. As such, we have several input modes for work that we do:
When a software developer says "I would like to be able to do X", we in Release Engineering strive to be able to say in reply "OK, we already know how to do X, let us show you".
For example, no one expected that a public, shared test environment would be particularly valuable until people who are now working in Release Engineering began to make the beta cluster useful. For another example, shared development environments using Vagrant were only an experiment until people who are now working in Release Engineering made them robust and useful.
From time to time the Foundation mandates particular practices. Automated browser tests were a mandate in early 2012. A fast deploy cycle was a mandate in early 2013. Replacing Bugzilla with Phabricator was a mandate in 2014. People in Release Engineering were critical to the success of each of these projects.
Many of the teams we work with request improvements to the tools and systems that Release Engineering work with. We work to accommodate these requests.
Clearly, Release Engineering priorities can undergo large shifts in short amounts of time. In general, we prioritize reactions over mandates over anticipation. We serve our colleagues first.
But we must also always balance ease, safety, and speed. The easiest thing to do is not always safe or fast, and negotiating the work we do in the context of the values we share occurs daily, weekly, monthly, and quarterly amongst the members of the Release Engineering team, and with the teams we serve.
We have adopted a number of practices to serve this negotiation:
- well attended weekly team meeting for all of Release Engineering
- Roadmap and Deployments meeting for senior WMF staff
- participating in Scrum of Scrums
- institutional pair-programming sessions among RelEng team people
- institutional pair-programming sessions for RelEng people and colleagues in other teams
- Željko Filipin and others
- training sessions and public presentations
- significant contributions to code repositories across WMF, from extensions to core to operations
- significant interaction with a volunteer community who share our interests, including past projects with Google Summer of Code and Outreachy
- close coordination with the Operations team, on whom we depend for production services
- close coordination with Team Practices Group, with whom we share a common interest in cross-team process, practice, and methodology