Deployment tooling

Sprints
We plan to put the items below into Release Management and QA related sprints sometime between July and December 2013:

Relevant Bugzilla searches:
 * whiteboard:deploysprint-13
 * whiteboard:rmqa-2013
 * whiteboard:rmqa-2013 or deploysprint-13

git-deploy

 * tracking
 * - auditing salt scripts for completeness
 * - deal with dirty git fetches properly
 * - Deployment-prep deploys from master and uses a submodule with submodules
 * Questions:

monitoring / reporting

 * - Setup monitoring for beta cluster
 * - Better 500 error/PHP exception monitoring
 * - Scaler's stderr should be captured and logged
 * - create monitoring for the issue in that bug
 * Questions:
 * Where should this be stashed in our monitoring?
 * Better reporting out of Jenkins builds status
 * including exporting results from our third-party Jenkins
 * overall dashboard of all build activity

deployment script improvements

 * - Make updates atomic (e.g. symlink + directory move tricks or git-deploy?)
 * - Some improvements for the deployment scripts
 * - Reconciling the use of timestamps on Javascript files (rsync vs ResourceLoader vs git)
 * - resetUserTokens.php not usable on large wikis
 * Kill deployment hacks with fire - live hacks that are still applied as of 2013-05-16

multi-site awareness

 * Database config cleanup -- multisite awareness in MediaWiki
 * can we fail over database?
 * Identify what on our side needs to be multi-site first.

Labs related

 * - the use of Beta cluster as a true canary (tracking)
 * - hermetic test environment on labs (vagrant or otherwise)
 * - setup monitoring of beta cluster moved under monitoring, ops stuff.
 * also usable for throw away dev environments for developer use