Deployment tooling/Cabal/2016-05-23

2016-05-23

Keyholder cleaning

• Keyholder cleanup
  • Ores somewhat blocked by this: https://gerrit.wikimedia.org/r/#/c/280403/51
  • TODO Test in beta :)
  • TODO Move public keys from various modules to secrets

Config deploy

• Puppet variable file on target
• Formatting of config template in repo
• Override

Deploying/testing other projects

• TODO doublecheck ':' in config values
• TODO phab ticket—local server deploying from local machine

Graph Deploy

• Example config: https://phabricator.wikimedia.org/P3167
  • this is a sample of phabricator's deployment config

List of Doom

Generated with:

find /srv/deployment -type d -maxdepth 2 | sort | nl (and some vim massage)

Services to migrate:

1. abacist/abacist
2.  Done analytics/aqs
3. analytics/kraken
4. analytics/refinery
5. cassandra/logstash-logback-encoder
6. cassandra/metrics-collector
7.  Done changeprop/deploy
8.  Done citoid/deploy
9.  Done cxserver/deploy
10. dropwizard/metrics
11. dumps/dumps
12. elasticsearch/plugins
13.  Done eventlogging/analytics
14.  Done eventlogging/eventbus
15. ??? eventlogging/eventlogging
16. fluoride/fluoride
17. gdash/gdash (deprecated)
18. grafana/grafana (deprecated, using debs now on krypton)
19. graphoid/deploy
20. iegreview/iegreview
21. integration/config
22. integration/kss
23. integration/mediawiki-tools-codesniffer
24. integration/php-coveralls
25. integration/phpcs
26. integration/phpunit
27. integration/slave-scripts
28. jobrunner/jobrunner
29. kartotherian/deploy
30. kibana/kibana
31. librenms/librenms [ops]
32. logstash/plugins
33.  Done mathoid/deploy
34.  Done mobileapps/deploy
35. ocg/ocg
36. parsoid/deploy
37.  Done phabricator/deployment
38. rcstream/rcstream
39. restbase/deploy
40. scholarships/scholarships
41. sentry/sentry
42. servermon/servermon [ops]
43. statsv/statsv
44. tessera/tessera (deprecated)
45. tilerator/deploy
46. wdqs/wdqs
47. wikimedia/discovery
48.  Done zotero/translation-server
49.  Done zotero/translators

by last fetch date:

tin:/srv/deployment$ ls -latr */*/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  117 Jul 16  2013 fluoride/fluoride/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  119 Feb 27  2014 integration/php-coveralls/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  118 Mar 19  2014 gdash/gdash/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev 4152 May 12  2014 parsoid/Parsoid/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  111 Aug 18  2014 integration/phpcs/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  259 Oct 16  2014 mathoid/mathoid/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  128 Dec  2  2014 restbase/deploy.old/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  121 Dec 11  2014 integration/mediawiki-tools-codesniffer/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  124 Dec 27  2014 abacist/abacist/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  113 Feb 19  2015 integration/phpunit/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  132 Feb 24  2015 dropwizard/metrics/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  120 May  7  2015 kibana/kibana/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  121 May 30  2015 rcstream/rcstream/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  121 Jun 30  2015 tessera/tessera/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  130 Aug 11  2015 logstash/plugins/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  134 Sep 17  2015 elasticsearch/plugins/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  113 Oct  2  2015 integration/slave-scripts/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  252 Oct  2  2015 grafana/grafana/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  113 Oct 22  2015 iegreview/iegreview/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  127 Dec  3 14:52 restbase/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  135 Dec 15 01:59 jobrunner/jobrunner/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  126 Jan  1 19:27 scholarships/scholarships/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  138 Mar 22 16:31 cassandra/logstash-logback-encoder/.git/FETCH_HEAD
-rw-rw-r-- 1 twentyafterfour wikidev  531 Mar 24 00:27 phabricator/deployment/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  226 Mar 31 20:16 eventlogging/eventlogging/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  138 Apr 15 13:32 zotero/translation-server/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  276 Apr 15 13:34 zotero/translators/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  123 Apr 24 20:02 statsv/statsv/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  125 Apr 26 09:08 citoid/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  360 Apr 26 13:52 dumps/dumps/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  215 Apr 27 07:00 servermon/servermon/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  122 Apr 27 16:06 librenms/librenms/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  252 Apr 28 17:19 kartotherian/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 mforns          wikidev  228 May  2 11:08 eventlogging/analytics/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  226 May  3 13:14 eventlogging/eventbus/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  130 May 10 12:26 mobileapps/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  271 May 10 17:39 ocg/ocg/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  128 May 10 17:58 graphoid/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  267 May 16 17:34 analytics/refinery/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  129 May 19 18:24 parsoid/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  115 May 19 21:00 tilerator/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  127 May 20 08:08 mathoid/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  138 May 20 08:15 changeprop/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  128 May 20 11:32 cxserver/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  141 May 23 11:09 cassandra/metrics-collector/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  887 May 23 17:05 wdqs/wdqs/.git/FETCH_HEAD

As Always

• Phase 2
• Phase 1
• Workboard https://phabricator.wikimedia.org/tag/scap3/
• Etherpad backed up to https://www.mediawiki.org/wiki/Deployment_tooling/Cabal
• Future document https://www.mediawiki.org/wiki/Deployment_tooling/Future
• Spreadsheet: https://docs.google.com/spreadsheets/d/1MlEsFxrLvdZdV_G82WEAIvBXr7ArO7nCEKaFClHhJEw/edit#gid=0