Wikimedia Release Engineering Team/Quarterly review, February 2014

Date:  | Time:  | Slides:  | Notes Who: Topics: Deploy process/pipeline, release process, bug fixing (the how of priority), code review, code management, security deploy/release, automation prioritization
 * Leads: Greg G, Chris M
 * Virtual team: Greg G, Chris M, Antoine, Sam, Chris S, Chad, Zeljko, Michelle G, Andre, Ariel
 * Other review participants (invited): Robla, Sumana, Quim, Maryana, James F, Ryan Lane, Ken, Terry, Tomasz, Alolita

Big picture
Release Engineering and QA are where our efforts in Platform can be amplified. When we do things well, we start to see more responsive development with higher quality code. That is our focus. What we want to accomplish: ...All in an effort to pave the path to a more reliable continuous deployment environment.
 * More appreciation of, response to, and creation of tests in development
 * Better monitoring and reporting out of our development and deployment processes, especially test environments and pre-deployment
 * Reduce time between code being merged and being deployed
 * Provide information about software quality in a way that informs release decisions
 * Help WMF Engineering learn and adapt from experience

Team roles
Many people outside of the virtual team play an important role in releases, but this review will focus on the work of the following people in the following roles:
 * Release engineering: Greg G, Sam, Chris S (security), Bryan Davis
 * QA and Test Automation: Chris M, Zeljko, Jeff Hall
 * Bug escalation: Andre, Greg G., Chris M, Chris S (security)
 * Beta cluster development/maintenance:' Antoine, Sam
 * Development tools (e.g. Gerrit, Jenkins): Antoine

Goals from last quarter
Keep: New Priority: Deprioritize:
 * Successfully managed the first release of MediaWiki in conjunction with our outside contractor - done
 * Work with Antoine on swift backed download.wikimedia - done
 * Browser tests managed in feature repos with feature teams - mostly done
 * more comprehensive quarter assessments of postmortems - in-progress
 * Automated API integration tests in important areas
 * UploadWizard - Done
 * Parsoid - Done
 * ResourceLoader - is a goal for Feb.
 * Expose fatal errors from both unit tests and browser tests to teams - Logstash in beta/production and the fatal monitoring script on Beta
 * Create process documentation for ideal test/deployment steps (eg: ThoughtWorks exercise) - done
 * Successfully streamline the review and deployment of extensions (Done in Q1) - still done
 * Manage build times, parallel execution, continuous execution of browser tests for optimum coverage (vague, ongoing goal) - no
 * Focus on community contributions and non-WMF - not really

Goals
vis a vis the WMF Engineering 2013-14 goals. Deployment Tooling Hiring Browser tests: target dev environments with bare wikis/one off instances//vagrant/"hermetic" test environments
 * Process through pain points from the Dev/Deploy review session
 * Need to document the feature requirements of sartoris/etc
 * Security Patch management
 * Complete hiring and train new Test Infrastructure Engineer
 * Complete hiring and train new QA Automation Engineer
 * in support of browser tests on WMF Jenkins
 * requires thoughtful use of the API

Dependencies
Ops dependency: MW Core dependency: Vagrant
 * Provide true HTTPS support on the Beta Cluster - not done
 * Incinga for Beta Cluster - done, but not yet accurate
 * https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep
 * http://ganglia.wmflabs.org/latest/?c=deployment-prep
 * http://icinga.wmflabs.org/cgi-bin/icinga/status.cgi?hostgroup=deployment-prep&style=detail
 * automation of deployment process
 * Monitoring of deploys (performance)
 * perf of scap - done
 * perf of certain parts of production services - in-progress
 * Security patch management on cluster (especially after new wmfXX branches are cut)
 * not yet addressed, on plate for next quarter
 * still need to determine maintainership
 * check in with Ori

Questions

 * to fill in