Wikimedia Release Engineering Team/Process

From mediawiki.org

Who we are[edit]

This is the Release Engineering team.

Mission[edit]

We get software from the developer to production as quickly, easily, and safely as possible.

We are big fans of the values of the Agile Manifesto:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

Our values

Quick

Software needs to be in front of users for it to have value. The longer code spends in a development environment, a test environment, a deployment queue, the more it costs and the less value it has. Release Engineering is in the business of making that process as fast as it can reasonably be made.

Easy

People creating software for users should have all the tools necessary right to hand, without having to struggle with tasks and chores unrelated to development. From planning tools and shared development environments and code repositories to test environments to deployment processes, Release Engineering works to make the steps from idea to users as easy and as automatic as they can be made.

Safe

Speed and ease are important, but not at the cost of quality. Release Engineering includes testing processes of all kinds, from code linters to continuous integration to automated system tests to production scripting. Furthermore, we take Quality Assurance seriously, in the sense that QA has to do with improvements to methodology and process throughout the development cycle.

Development process[edit]

Early in the dev process ---> ---> After merge to master branch ---> ---> Production
Planning & Tracking Shared Development Environment Continuous Integration Beta Cluster Browser Test Automation Browser Test Automation Exploratory Testing Exploratory Testing Deploying to Production Management
Q/E/S Easily Quickly/Easily Safely Safely Safely Safely Safely/Easily Safely/Easily Easily Quickly/Easily
Person Mukunda Modell Dan Duvall Antoine Antoine Željko Chris Elena Rummana Chad Greg
Tools and systems Phabricator Vagrant, Puppet Jenkins, Zuul Puppet, sysadmin page-object, Jenkins job builder page-object, Jenkins, Sauce Labs, beta cluster beta cluster, test2wiki Gerrit, Phabricator, beta cluster, test2wiki and enwiki Gerrit, scap SWAT, logs, schedules
Activities Maintenance, new features in response to dev teams' requests. Maintenance, improvements to code and infrastructure. Maintenance, new features Maintenance, updates to improve performance and emulation of prod systems Jenkins infrastructure, shared code, community training Failure analysis, test design, internal training, process Exploratory testing, reporting issues, bug triage Exploratory testing, reporting issues, bug triage,bug verification. Weekly deployments, SWAT when necessary Oversee production deployments, do schedules, planning
Collaborates with Ops, ECT Core, Features teams, browser test Core, Ops, Labs, Devs, Volunteers Ops, Labs Language, Search, Wikidata, ECT Mobile, VE, Flow, MMV, Labs, Team practices Editing Editing and Mobile Apps team Core, Ops Everyone
Working on Expand Phab capabilities Vagrant in CI, more shared code Performance, browser tests voting, CI infra Better prod emulation, staging Browser tests voting
  • Refactoring; EAL
  • Vagrant in Jenkins plan
  • Process work (with TeamPractices)
ET for other WMF teams; help with automated tests ET for other WMF teams,making browser tests complete and reliable for high-priority features in VE for Q3 ,helping with bug triage More automation More automation, more information

Tools and Frameworks[edit]

Release Engineering oversees many of the tools and frameworks that help in the delivery of software features, from development environments and test frameworks to deployment scripts. Creating and maintaining reliable automation is a constant concern for us on every front.

Here is a list of tools and frameworks that people in Release Engineering understand and maintain. We may not be the only ones who know about each item, but we are all knowledgeable to a certain extent about all of the systems listed here.

Initial Tools and frameworks People with answers
B Browser tests Chris, Željko, Rummana
BC Beta cluster Chad, Mukunda, Tyler, Chris, Antoine
ET Exploratory Testing Rummana, Elena (Željko, Chris also)
J Jenkins/Zuul Antoine, Željko, Tyler
P Puppet Dan, Mukunda
PH Phabricator Mukunda
SC SCAP Greg, Mukunda, Chad
SL SauceLabs Željko, Chris
SW SWAT Greg
V Vagrant Dan

Teams and Tools[edit]

Release Engineering has direct, reciprocal relationships with at least ten other teams within WMF. Here is a table that attempts to capture the nature of those relationships in terms of the tools and frameworks that we use in common.

Team Tools
Editing B BC ET J PH SC SL SW V
Mobile Web/Apps B BC ET J PH SC SL SW V
Services/Parsoid BC J PH SW
Collaboration B BC J PH SC SL SW V
Language B BC J PH SC SL V
Core BC J P PH SC SW
Multimedia B BC J PH SC SL SW V
Engineering Community Team B(?) BC PH
Team Practices Group PH
Ops BC P SC
(Fundraising)
(Zero)
(Design)

How Release Engineering Works[edit]

Release Engineering creates, manages, and supports the tools and practices that allow developers to create software easily, manage software safely, and get software to users quickly. As such, we have several input modes for work that we do:

Anticipation[edit]

When a software developer says "I would like to be able to do X", we in Release Engineering strive to be able to say in reply "OK, we already know how to do X, let us show you".

For example, no one expected that a public, shared test environment would be particularly valuable until people who are now working in Release Engineering began to make the beta cluster useful. For another example, shared development environments using Vagrant were only an experiment until people who are now working in Release Engineering made them robust and useful.

Mandate[edit]

From time to time the Foundation mandates particular practices. Automated browser tests were a mandate in early 2012. A fast deploy cycle was a mandate in early 2013. Replacing Bugzilla with Phabricator was a mandate in 2014. People in Release Engineering were critical to the success of each of these projects.

Reaction[edit]

Many of the teams we work with request improvements to the tools and systems that Release Engineering work with. We work to accommodate these requests.

Setting priorities[edit]

Clearly, Release Engineering priorities can undergo large shifts in short amounts of time. In general, we prioritize reactions over mandates over anticipation. We serve our colleagues first.

But we must also always balance ease, safety, and speed. The easiest thing to do is not always safe or fast, and negotiating the work we do in the context of the values we share occurs daily, weekly, monthly, and quarterly amongst the members of the Release Engineering team, and with the teams we serve.

We have adopted a number of practices to serve this negotiation:

  • well attended weekly team meeting for all of Release Engineering
  • Roadmap and Deployments meeting for senior WMF staff
  • participating in Scrum of Scrums
  • institutional pair-programming sessions among RelEng team people
  • institutional pair-programming sessions for RelEng people and colleagues in other teams
    • Željko Filipin and others
  • training sessions and public presentations
  • significant contributions to code repositories across WMF, from extensions to core to operations
  • significant interaction with a volunteer community who share our interests, including past projects with Google Summer of Code and Outreachy
  • close coordination with the Operations team, on whom we depend for production services
  • close coordination with Team Practices Group, with whom we share a common interest in cross-team process, practice, and methodology