MediaWiki 1.19/Post-deploy retrospective

We deployed MediaWiki 1.19 in February 2012. In the interests of doing better next time, please help us brainstorm what went right and what we need to look at. RobLa will then summarize that discussion.

Note: this is specifically about the MediaWiki 1.19 deployment, rather than the 1.19.0 release.

There are two sections: "What went well?" and "What do we need to look at?" For now, limit input to small (<140 character) bullet items. If you want to elaborate or respond, please take it to the Talk page.

What went well?

 * Pre-deploy testing of Commons gadgets on http://labs.wikimedia.beta.wmflabs.org/
 * Gross errors detected quickly, rollbacks were effective and fast.
 * Most of the big bugs got shaken out prior to the *.wikipedia.org deployment
 * Detailed communication plan, well in advance
 * No deathmarches. A couple of the windows went over their time, but they weren't huge marathons.
 * Most schema changes were done well in advance
 * DB changes were back compatible ensuring smooth migration and easy rollback
 * HetDeploy works!
 * Less new features deployed meant less bug
 * Deployed nlwiki at the exact right time
 * High quality deployment
 * Communications on the village pumps
 * Proofread page extension testing
 * Schedule in advance
 * Just in time branching
 * and then code slush!!!! saved us from merge conflict
 * Chad was busy with git and not around to bother anyone

What do we need to look at?

 * Swift?
 * We misread the severity of broken thumb reports
 * Release notes were unorganized, hard to interpret, possibly inaccurate
 * Editors asked more than once "What has changed" and it was hard to give an answer. MediaWiki 1.18 did this much better.
 * Should we have a watchlist notice on top wikis?
 * Or all wikis? We had a bug of a user complaining they hadn't heard anything about it...(Every user talk page on every wiki)
 * https://bugzilla.wikimedia.org/show_bug.cgi?id=34704
 * Lots of Javascript breakage that we might have been able to test sooner
 * Need to test language variants prior to deployment
 * Need to make sure we're not scheduling right on top of key personnel vacation (1.19: Asher)
 * Possible to do schema changes even earlier or independent of deploy cycle?
 * IRC push notifications were broken because of log formatting rewrite.
 * Status of Messages*.php from 1.18 vs 1.19 is confusing
 * User-visible feature communication (diffs this time around, math formulae in PNG)
 * Namespace changes
 * Proofread page extension testing - this extension needs someone to love & rewrite it
 * Unmaintained extensions
 * Code review
 * 1.19 bugzilla milestones/tracking/etc (confusion)
 * Signaling on the roadmap page - more updates
 * Roadmap page should be marked more authoritative
 * Find out a way to not block volunteers developer while deploying (codeslush)
 * Some extensions not updated for FileBackend