Wikimedia Release Engineering Team/Wishlist

This is the wishlist for the Release Management and QA team.

Code Deploy Dashboard
Problem: We don't have an simple way of communicating what code is deployed to which wikis. Right now you need a combination of Gerrit and the MediaWiki Roadmap.

Stakeholders/use-cases

 * Product Managers: I want to see one page where I can tell if code my team has written is on all Wikipedias.
 * Developers: I want to know when code I have written is on the test wikis and when it'll be on English Wikipedia.
 * Technical Users/Readers: I want to see if there's an obvious excuse for a problem I am seeing on my home wiki due to a recent deployment.
 * Release Team: All of the above.

Solution ideas
Create a dashboard page that displays the currently deployed/active branches on the WMF Cluster (and, for bonus points, the Beta Cluster). The branches will link to a single page in Gerrit that lists all of the changes and their commit short message.

See this quick/"I'm not a designer" mockup from Greg.

Pieces needed

 * Timestamp of deploys (and which group of wikis was affected, eg phase0 or phase1) (from graphite/statsd if that ever works again )
 * A way to associated a wiki group with the version that's live (see hashar's work on Special:Version parsing?)
 * Super basic but informative graphs (probably something pulled from http://gdash.wikimedia.org/dashboards/reqerror/ )
 * A place to put that all

Release Notes Sanity
Problem: 13:09 <    bd808> Gah! Release notes! F U release notes 13:10 <   greg-g> :) 13:10 <     ori-l> it's one of our "you forgot to say simon says" -1 tricks

Stakeholders:
 * All MediaWiki developers - When commiting a change to MW that needs a note in the RELEASE-NOTES-X.XX file, I shouldn't have to play games with gerrit and rebasing continuously.

Solution ideas:
 * rm -f RELEASE-NOTES-X.XX
 * Merge conflict resolution driver

True code pipeline
Right now code does not follow the same pipeline universally. For example:
 * Code merged to master on Friday will be on the Beta Cluster for one full week before going out on the testwikis
 * Code merged to master on Thursday early morning will be on the Beta Cluster for all of an hour (or a minute) before going to testwikis.

This is not ok. Code should propagate from master -> Beta Cluster -> test production -> other wikis in a consistent fashion.

Proposal
Nightly test tags - tracking:


 * Every night at a specified time we tag master in core and all extensions in use on the cluster.
 * pre/post merge unit tests are run against the tag
 * the tag is deployed to the Beta Cluster using multiversion -
 * browser tests are run against the beta cluster hitting the nightly tag version -
 * in the morning we review the browser tests and unit test output, determine suitability for deploy
 * deploy the tag as the new wmfXX deployed version on Thursday morning
 * If the latest tag is not suitable (due to failed browser tests etc) we use the last suitable nightly (eg Tues night's)

Auto-populate 'Important Changes' from RELEASE-NOTES-1.XX
See the 'important changes' section in most wmfXX release pages, eg this one.

It would be nice to auto-create that based on the RELEASE-NOTES file.

See the RFC: Requests_for_comment/Release_notes_automation

block commits on warnings... almost
Let's make higher quality code. One way is to have a voting test that fails on code compile/build warnings. That's a bit extreme.

An alternative is to compare the number of warnings (and TODOs/FIXMEs?) before and after the commit and only fail if that number increases.

Performance testing environment
See bug 65394