Deployment tooling/Notes/New branch deploy

See wikitech:Heterogeneous deployment/Train deploys for live documentation of the Wikimedia deployment process for MediaWiki.

Typical process for deploying a new branch of MediaWiki to the wiki[mp]edia production cluster.

Overview

 * Create a new 1.XwmfN branch from current HEAD
 * Copy that new version to all MW servers
 * Change all Wikipedias to use 1.XwmfN-1 branch
 * Change group0 wikis (test/test2/testwikidata/mediawiki) to use 1.XwmfN branch
 * Update documentation

Detailed process

 * 1) Run make-wmf-branch script
 * 2) Branch all extensions
 * 3) Branch core
 * 4) Update core branch submodules to reference extension branch HEADs
 * This could be done by a Jenkins job or similar automated process
 * Should involve a gate process that chooses a stable branch point rather than whatever the state of each HEAD is at the time the process is run
 * Should be tested as a unit by regression tests before going to tin
 * 1) Run checkoutMediaWiki on tin
 * 2) Checkout branch in a new directory on tin
 * 3) Initialize submodules
 * 4) Create symlinks to new branch from bits and w to static assets in branch
 * 5) Apply any unmerged security patches from N-1 branch
 * Should be automated an incorporated into checkout script
 * Should be made to be easier to audit to validate that all patches are present, especially for submodules
 * 1) Git add and commit symlinks
 * 2) Clean up branches on deploy server that have been inactive for >31 days
 * 3)  Cleanup bits symlinks to branch
 * 4) * /a/common/multiversion/deleteMediaWiki php-1.23wmfX
 * 5) Create symlinks cleanup patch
 * 6) * git rm -r docroot/bits/static-1.23wmfX
 * 7) * git rm -r w/static-1.23wmfX
 * 8) * NOLOGMSG=1 git commit -m 'Remove 1.23wmfX symlinks'
 * 9) Revert change on tin
 * 10) * NOLOGMSG=1 git reset HEAD^ --hard
 * 11) Repeat steps above for each branch as needed
 * 12) Run updateWikiversions
 * 13) Change wikiversions.json to run all on version N-1
 * 14) Git add wikiversions.json and git commit
 * 15) Run updateWikiversions
 * 16) Change wikiversions.json to run group0 on version N
 * 17) Git add wikiversions.json and git commit
 * 18) Git push symlink and wikiverisons patches to gerrit
 * 19) Git reset HEAD~4 working directory and index on tin
 * Should be scripted
 * Should not require ssh agent forwarding on tin
 * 1) Review and approve the symlinks patches in gerrit
 * 2) Wait for zuul & Jenkins to merge patch to branch
 * 3) Git fetch --rebase working copy on tin
 * 4) Delete expired branch checkout in /a/common
 * 5) * rm -rf /a/common/php-1.23wmfX
 * 6) * Repeat steps above for each branch as needed
 * 7) Manually edit wikiverisons.json to run testwiki on version N
 * 8) Run scap
 * 9) Validate PHP syntax in config directory
 * 10) Copy staging directory to common-local on tin (for utility scripts)
 * 11) Update l10n cache for version N-1
 * 12) Use PHP to read all l10n files and create CDB for each language
 * 13) Create json copies of CDB contents
 * 14) Create MD5 checksums of json copies
 * 15) Build l10n cache for version N
 * 16) Use PHP to read all l10n files and create CDB for each language
 * 17) Create json copies of CDB contents
 * 18) Create MD5 checksums of json copies
 * 19) Copy state of tin to sync slaves via rsync pull
 * 20) Copy state of sync slaves to all MW servers via rsync pull
 * 21) Build l10n cache on all WM servers
 * 22) Compile wikiverisons.json to cdb
 * 23) Copy wikiverions.{json,cdb} to all MW servers
 * 24) Revert local changes to wikiverisons on tin
 * 25) Test testwiki & watch for cluster errors
 * 26) Review and approve the all wikis on version N-1 patch
 * 27) Wait for zuul & Jenkins to merge patch to branch
 * 28) Git fetch --rebase working copy on tin
 * 29) Run sync-wikiverisons
 * 30) Compile wikiverisons.json to cdb
 * 31) Copy wikiverions.{json,cdb} to all MW servers
 * 32) Test wikis & watch for cluster errors
 * 33) Review and approve the group0 on version N patch
 * 34) Wait for zuul & Jenkins to merge patch to branch
 * 35) Git fetch --rebase working copy on tin
 * 36) Run sync-wikiverisons
 * 37) Compile wikiverisons.json to cdb
 * 38) Copy wikiverions.{json,cdb} to all MW servers
 * 39) Test group0 & watch for cluster errors
 * 40) Create/update deploy notes on mw.o
 * 41) git checkout --track -b wmf/N origin/wmf/N
 * 42) git submodule update --init --recursive
 * 43) php uploadChangelog.php wmf/N
 * 44) git checkout --track -b wmf/N-1 origin/wmf/N-1
 * 45) git submodule update --init --recursive
 * 46) php uploadChangelog.php wmf/N-1
 * 47) git checkout --track -b wmf/N-2 origin/wmf/N-2
 * 48) git submodule update --init --recursive
 * 49) php uploadChangelog.php wmf/N-2
 * This should be automated to run every day or after every branch merge
 * Should use a bot account for upload
 * Upload script should use oauth for authentication
 * 1) Update Roadmap with deploy status