Heterogeneous deployment/etherpad

''This is a copy/paste of the content on HetDeploy etherpad page for posterity/safe keeping. The last substantive edit to that etherpad was made in July 2011.''

Requirements Nice to have Issues: 2011-07-13 http://svn.wikimedia.org/viewvc/mediawiki/trunk/debs/wikimedia-task-appserver/ Directory layout * common * common/php-1.17 * common/wmf-config Priyanka's work: Example: http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/mwmultiversion/scripts/wikiversions.dat.sample Stuff to do: Roan: need a script to create a new version? Tim: we could do that Tim: other thing that's not written: command line wrapper. Need something that lets you run the command line scripts relative to the maintenance directory in the relevant version-specific directory * php mwmaintenance.php somescript.php --wiki=enwiki ...args... * with shell script wrapper? /usr/bin/mwmaint somescript.php --wiki=enwiki ..args... * Support out-of-tree maintenance scripts by running them with the MW_INSTALL_PATH environment variable set appropriately * Support extension maintenance scripts and core maintenance scripts in strange directories like includes/normal Need to handle compiled binaries like texvc. Already have the script for creating a version specific directory, but needs to be set up to use it, and compile multiple versions. Another thing to think about: how to deploy these changes without breaking the site. For moving wfconfig Also, swap in the maintenance wrapper Update crontabs running maintenance scripts Clean up all uses of the php symlink. The concept of the "real default version" is going to go away. Tim: there are uses for default version. For example, we have old legacy URLs. It might make sense to push all of those to the latest version. General status-quo notes: Other notes: $wgLocalStylePath needed for csshover.htc (same origin) Next steps: How to populate /usr/local/apache/common/wikiversions.db8 Backlog: Plan for moving wmf-config: Partial deployment 1: Partial deployment 2A: Partial deployment 2B: Partial deployment 2C: Partial deployment 3: Partial deployment 4:
 * Multiple core versions
 * Multiple extension versions
 * Ability to have at least staging + production
 * A/B testing
 * bits: $wgExtensionAssetsPath and $wgStyleSheetPath
 * l10n cache
 * separate $IP, but shared configuration
 * scap pushing multiple versions
 * downgrading
 * (figure out if there are any other shared caches)
 * APC cache size
 * patching CommonSettings.php
 * web wrapper - for 1.17 we had mwversion.php. She was working on extending it to work with CDB. There's a text file with a list of versions, and then there's a script to compile it into a cdb file, and that gets pushed with scap, and then that file controls which version gets used
 * To do: clear up confusion between version names and directory names
 * Avoid reconfiguring Apache. Use symlinks and sync that out?
 * Change the directory layout. may just want to keep hyphens for now
 * copy to ../wmf-config
 * change CommonSettings.php to point to ../wmf-config
 * delete old wmf-config
 * home/wikipedia/common - "where everything lives"
 * sync mirrors this stuff to local/apache/common-local
 * p/php/php-1.17/php-1.5/wmf-deployment -> php-1.17
 * common/live-1.5/ has entry-wrappers
 * skins-1.17/1.5 end up at 1.17 for b/c (via symlinks)
 * aliasing config in home/wikipedia/conf/httpd/wikimedia.conf ()
 * LU updates L10n files, syncs them, LU notices timestamp change and updates tmp/cache files
 * Use docroot/bits for new bits symlinks?
 * /tmp/mw-cache cache not commonized
 * Roan and Aaron have a walkthrough
 * A helper script to do this is checked into scripts/cdbmake-12.sh9
 * A sample file that contains input for cdbmake-12.sh is checked into scripts/wikiversions.dat.sample
 * Set $IP in MWVersion
 * Maintenance script wrappers
 * Multi-versioned compiled binaries (scap2)
 * LU needs to run for each version
 * Possible rewrite of MWMultiVersion.php (clean out legacy cruft)
 * Script for creating a new MW version from an svn url
 * In PHP?
 * Does SVN checkout of mediawiki/branches/wmf/1.XXwmfX/
 * Creates LocalSettings.php and symlink to AdminSettings.php/StartProfiler.php
 * Bits doc root symlinks
 * addwiki.php needs to populate /usr/local/apache/common/wikiversions.db8
 * deal with ExtensionMessages.php/scap
 * Deployment: MFT r92422
 * Deployment: Move wmf-config up a directory
 * Deployment: Install MWMultiVersion & maintenance script wrappers to cluster
 * Deployment: populate wikiversions.cdb from all.dblist
 * Deployment: Update crontabs (and .sh files) for running maintanence scripts
 * http://wikitech.wikimedia.org/view/Cron_jobs
 * Deployment: Integrate CommonSettings changes
 * Deployment: Deploy MW 1.18 to test2
 * wgCacheDirectory should be moved CommonSettings.php and versioned
 * Tell everyone that they can't touch wmf-config while this is going on
 * cp -a /home/wikipedia/common/php/wmf-config /home/wikipedia/common/wmf-config and fix perms as required
 * Edit /h/w/c/wmf-config/CommonSettings.php to set $wmfConfigDir correctly (to /h/w/c/wmf-config instead of /h/w/c/php/wmf-config)
 * Run scap, pushing the dead code in /h/w/c/wmf-config to the Apaches
 * Edit /h/w/c/php/LocalSettings.php to point to the new location of wmf-config
 * Sync LocalSettings.php . Site now uses new dir
 * Clean up other references to old dir (in doMaintenance.php, right?) and kill the old dir
 * checkout new common/multiversion scripts
 * run populateWikiversionCDB.php and test it
 * install & test mwversionsinuse (test --withdb too)
 * scap
 * rename live MWVersion.php to MWVersion-old.php
 * modifiy MWVersion.php to use the new one for testwiki and the old one for all else
 * scap
 * test around with testwiki (web views)
 * cdb stuff broken -- fixme
 * fix MWScript scoping
 * copy ExtensionMessages.php to ExtensionMessages-1.17.php
 * deploy Maintenance.php fix
 * make all wikis use the new MWVersion.php
 * www.wikipedia.org portal broke - fix
 * change scap to create ExtensionMessages-1.17.php and CommonSettings to use it
 * scap
 * update addwiki.php and sync
 * rename live CommonSettings.php to CommonSettings-old.php
 * install new CommonSettings.php (merge changes first)
 * test around with testwiki (web views)
 * test MWScript.php with eval
 * scap
 * add logging for maintenance scripts called without wrapper?
 * install & test mwscript
 * Look in usr/bin, wikipedia/bin
 * Change jobs-loop.sh
 * Look at http://wikitech.wikimedia.org/view/Cron_jobs
 * Look at crontabs in etc/ on hume?
 * mw-central-notice, mw-flagged-revs, mw-tor-list
 * Look at crontabs in etc/ on formey?
 * maintenance/mwdocgen.php ok (doesnt use wmf stuff, standard install)
 * Look at everything mounting /home (dumps)
 * Ask Ryan and Ariel
 * scap
 * deploy extended version number code
 * copy scap-2 changes to debs/wikimedia-task-appserver in SVN
 * svn update scap-2
 * install l10nupdate and sync-l10nupdate changes
 * scap
 * profit!