Deployment tooling/Cabal/2015-06-08

June 8th

 * Created tickets to track discussion
 * https://phabricator.wikimedia.org/T101023


 * conversation between Mukunda, Bryan: possibility of using git in scap deploy by buildling local repo from core + extensions
 * Middle-step git repository for mediawiki
 * binaries/blobs?
 * might be possible with git-annex, better than git fat
 * git-torrent? might be worth further research https://github.com/cjb/GitTorrent
 * how do we handle submodules?
 * building local repo for deployment might ... ? (sorry, missed it)
 * Current services uses git via apache
 * Sometimes subjectively slow, may be room for improvement
 * Fan out
 * Preseeding, proxies needed -- could this be a post-merge jenkins job?
 * Deploy master node
 * Currently always tin
 * Could  start deploy from other proxies (pre-seed target)
 * Agnosticly built in terms of proxies vs pre-seed target
 * Might be good time to experiment with git based transport
 * Determine whether it's going to be over ssh/https/torrent/etc.
 * Run tests outside of prod to determine if feasible for MW
 * nginx/varnish, something, in each datacenter for fanout proxy
 * less clobbering
 * New work_dir after fetch with filecheckout

*** Pros: **** Scalable: github uses it  **** Supported: seems to be developed and supported by some of the core git team *** Reliability (should check out the actual file not just text of sha) *** Flexibility of transport mechanism *** Scalability and reprudicibiliity (scales up and down) *** Resource consuption **** CPU **** Limit network IO    **** Limit disk IO    *** Speed *** Could possibly simulate cross-datacenter limitations with `tc` *** eqiad - codfw RTT ~40ms, inside eqiad RTT is ~0.5ms
 * Moving git blobs solutions
 * Git large file store https://git-lfs.github.com/
 * Git Annex
 * GitTorrent
 * Git Fat
 * Criteria:


 * Definitions of success
 * Atomic failure modes
 * Optional fanout proxies based on git-http(s)
 * Blobs pused around with git annex or another solution (GitTorrent, large file store, figure out criteria for solution)
 * Continuous feedback to user
 * Verify integrity of repo
 * Not insanely slow (any slower than current system)

To use staging for deployment blob mechanisms