Deployment tooling/Cabal/2016-05-23

Keyholder cleaning

 * Keyholder cleanup
 * Ores somewhat blocked by this: https://gerrit.wikimedia.org/r/#/c/280403/51
 * TODO Test in beta :)
 * TODO Move public keys from various modules to secrets

Config deploy

 * Puppet variable file on target
 * Formatting of config template in repo
 * Override

Deploying/testing other projects

 * TODO doublecheck ':' in config values
 * TODO phab ticket—local server deploying from local machine

Graph Deploy

 * Example config: https://phabricator.wikimedia.org/P3167
 * this is a sample of phabricator's deployment config

List of Doom
Generated with: find /srv/deployment -type d -maxdepth 2 | sort | nl (and some vim massage)

Services to migrate:
 * 1) abacist/abacist
 * 2) ✅ analytics/aqs
 * 3) analytics/kraken
 * 4) analytics/refinery
 * 5) cassandra/logstash-logback-encoder
 * 6) cassandra/metrics-collector
 * 7) ✅ changeprop/deploy
 * 8) ✅ citoid/deploy
 * 9) ✅ cxserver/deploy
 * 10) dropwizard/metrics
 * 11) dumps/dumps
 * 12) elasticsearch/plugins
 * 13) ✅ eventlogging/analytics
 * 14) ✅ eventlogging/eventbus
 * 15) ??? eventlogging/eventlogging
 * 16) fluoride/fluoride
 * 17) gdash/gdash (deprecated)
 * 18) grafana/grafana (deprecated, using debs now on krypton)
 * 19) graphoid/deploy
 * 20) iegreview/iegreview
 * 21) integration/config
 * 22) integration/kss
 * 23) integration/mediawiki-tools-codesniffer
 * 24) integration/php-coveralls
 * 25) integration/phpcs
 * 26) integration/phpunit
 * 27) integration/slave-scripts
 * 28) jobrunner/jobrunner
 * 29) kartotherian/deploy
 * 30) kibana/kibana
 * 31) librenms/librenms [ops]
 * 32) logstash/plugins
 * 33) ✅ mathoid/deploy
 * 34) ✅ mobileapps/deploy
 * 35) ocg/ocg
 * 36) parsoid/deploy
 * 37) ✅ phabricator/deployment
 * 38) rcstream/rcstream
 * 39) restbase/deploy
 * 40) scholarships/scholarships
 * 41) sentry/sentry
 * 42) servermon/servermon [ops]
 * 43) statsv/statsv
 * 44) tessera/tessera (deprecated)
 * 45) tilerator/deploy
 * 46) wdqs/wdqs
 * 47) wikimedia/discovery
 * 48) ✅ zotero/translation-server
 * 49) ✅ zotero/translators

by last fetch date:

tin:/srv/deployment$ ls -latr */*/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  117 Jul 16  2013 fluoride/fluoride/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  119 Feb 27  2014 integration/php-coveralls/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  118 Mar 19  2014 gdash/gdash/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev 4152 May 12  2014 parsoid/Parsoid/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  111 Aug 18  2014 integration/phpcs/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  259 Oct 16  2014 mathoid/mathoid/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  128 Dec  2  2014 restbase/deploy.old/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  121 Dec 11  2014 integration/mediawiki-tools-codesniffer/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  124 Dec 27  2014 abacist/abacist/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  113 Feb 19  2015 integration/phpunit/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  132 Feb 24  2015 dropwizard/metrics/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  120 May  7  2015 kibana/kibana/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  121 May 30  2015 rcstream/rcstream/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  121 Jun 30  2015 tessera/tessera/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  130 Aug 11  2015 logstash/plugins/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  134 Sep 17  2015 elasticsearch/plugins/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  113 Oct  2  2015 integration/slave-scripts/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  252 Oct  2  2015 grafana/grafana/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  113 Oct 22  2015 iegreview/iegreview/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  127 Dec  3 14:52 restbase/deploy/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  135 Dec 15 01:59 jobrunner/jobrunner/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  126 Jan  1 19:27 scholarships/scholarships/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  138 Mar 22 16:31 cassandra/logstash-logback-encoder/.git/FETCH_HEAD -rw-rw-r-- 1 twentyafterfour wikidev 531 Mar 24 00:27 phabricator/deployment/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  226 Mar 31 20:16 eventlogging/eventlogging/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  138 Apr 15 13:32 zotero/translation-server/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  276 Apr 15 13:34 zotero/translators/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  123 Apr 24 20:02 statsv/statsv/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  125 Apr 26 09:08 citoid/deploy/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  360 Apr 26 13:52 dumps/dumps/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  215 Apr 27 07:00 servermon/servermon/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  122 Apr 27 16:06 librenms/librenms/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  252 Apr 28 17:19 kartotherian/deploy/.git/FETCH_HEAD -rw-rw-r-- 1 mforns         wikidev  228 May  2 11:08 eventlogging/analytics/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  226 May  3 13:14 eventlogging/eventbus/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  130 May 10 12:26 mobileapps/deploy/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  271 May 10 17:39 ocg/ocg/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  128 May 10 17:58 graphoid/deploy/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  267 May 16 17:34 analytics/refinery/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  129 May 19 18:24 parsoid/deploy/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  115 May 19 21:00 tilerator/deploy/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  127 May 20 08:08 mathoid/deploy/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  138 May 20 08:15 changeprop/deploy/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  128 May 20 11:32 cxserver/deploy/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  141 May 23 11:09 cassandra/metrics-collector/.git/FETCH_HEAD -rw-rw-r-- 1 trebuchet      wikidev  887 May 23 17:05 wdqs/wdqs/.git/FETCH_HEAD

As Always

 * Phase 2
 * Phase 1
 * Workboard https://phabricator.wikimedia.org/tag/scap3/
 * Etherpad backed up to https://www.mediawiki.org/wiki/Deployment_tooling/Cabal
 * Future document https://www.mediawiki.org/wiki/Deployment_tooling/Future
 * Spreadsheet: https://docs.google.com/spreadsheets/d/1MlEsFxrLvdZdV_G82WEAIvBXr7ArO7nCEKaFClHhJEw/edit#gid=0