Deployment tooling/Notes/New branch deploy

Typical process for deploying a new branch of MediaWiki to the wiki[mp]edia production cluster.

Overview

 * Create a new 1.XwmfN branch from current HEAD
 * Copy that new version to all MW servers
 * Change all Wikipedias to use 1.XwmfN-1 branch
 * Change group0 wikis (test/test2/testwikidata/mediawiki) to use 1.XwmfN branch
 * Update documentation

Detailed process

 * 1) Run make-wmf-branch script
 * 2) Branch all extensions
 * 3) Branch core
 * 4) Update core branch submodules to reference extension branch HEADs
 * This could be done by a Jenkins job or similar automated process
 * Should involve a gate process that chooses a stable branch point rather than whatever the state of each HEAD is at the time the process is run
 * Should be tested as a unit by regression tests before going to tin
 * 1) Run checkoutMediaWiki on tin
 * 2) Checkout branch in a new directory on tin
 * 3) Initialize submodules
 * 4) Create symlinks to new branch from bits and w to static assets in branch
 * 5) Apply any unmerged security patches from N-1 branch
 * Should be automated an incorporated into checkout script
 * Should be made to be easier to audit to validate that all patches are present, especially for submodules
 * 1) Git add and commit symlinks
 * 2) Run updateWikiversions
 * 3) Change wikiversions.json to run all on version N-1
 * 4) Git add wikiversions.json and git commit
 * 5) Run updateWikiversions
 * 6) Change wikiversions.json to run group0 on version N
 * 7) Git add wikiversions.json and git commit
 * 8) Git push symlink and wikiverisons patches to gerrit
 * 9) Git reset HEAD~3 working directory and index on tin
 * Should be scripted
 * Should not require ssh agent forwarding on tin
 * 1) Review and approve the symlinks patch in gerrit
 * 2) Wait for zuul & Jenkins to merge patch to branch
 * 3) Git fetch --rebase working copy on tin
 * 4) Manually edit wikiverisons.json to run testwiki on version N
 * 5) Run scap
 * 6) Validate PHP syntax in config directory
 * 7) Copy staging directory to common-local on tin (for utility scripts)
 * 8) Update l10n cache for version N-1
 * 9) Use PHP to read all l10n files and create CDB for each language
 * 10) Create json copies of CDB contents
 * 11) Create MD5 checksums of json copies
 * 12) Build l10n cache for version N
 * 13) Use PHP to read all l10n files and create CDB for each language
 * 14) Create json copies of CDB contents
 * 15) Create MD5 checksums of json copies
 * 16) Copy state of tin to sync slaves via rsync pull
 * 17) Copy state of sync slaves to all MW servers via rsync pull
 * 18) Build l10n cache on all WM servers
 * 19) Compile wikiverisons.json to cdb
 * 20) Copy wikiverions.{json,cdb} to all MW servers
 * 21) Revert local changes to wikiverisons on tin
 * 22) Test testwiki & watch for cluster errors
 * 23) Review and approve the all wikis on version N-1 patch
 * 24) Wait for zuul & Jenkins to merge patch to branch
 * 25) Git fetch --rebase working copy on tin
 * 26) Run sync-wikiverisons
 * 27) Compile wikiverisons.json to cdb
 * 28) Copy wikiverions.{json,cdb} to all MW servers
 * 29) Test wikis & watch for cluster errors
 * 30) Review and approve the group0 on version N patch
 * 31) Wait for zuul & Jenkins to merge patch to branch
 * 32) Git fetch --rebase working copy on tin
 * 33) Run sync-wikiverisons
 * 34) Compile wikiverisons.json to cdb
 * 35) Copy wikiverions.{json,cdb} to all MW servers
 * 36) Test group0 & watch for cluster errors
 * 37) Create/update deploy notes on mw.o
 * 38) git checkout --track -b wmf/N origin/wmf/N
 * 39) git submodule update --init --recursive
 * 40) php uploadChangelog.php wmf/N
 * 41) git checkout --track -b wmf/N-1 origin/wmf/N-1
 * 42) git submodule update --init --recursive
 * 43) php uploadChangelog.php wmf/N-1
 * 44) git checkout --track -b wmf/N-2 origin/wmf/N-2
 * 45) git submodule update --init --recursive
 * 46) php uploadChangelog.php wmf/N-2
 * This should be automated to run every day or after every branch merge
 * Should use a bot account for upload
 * Upload script should use oauth for authentication
 * 1) Update Roadmap with deploy status