Wikimedia Release Engineering Team/Deployment pipeline/2019-02-28

Last Time

• 2019-02-14
• Archive

Current Quarter Goals

• Roughly a month left!

• TEC3:O6:O:6.1:Q3: Deployment Pipeline Documentation
  • cf: [#TODOs_from_last_time]

• TEC3:O3:O3.1:Q3: Move cxserver, citoid, changeprop, eventgate (new service) and ORES (partially) through the production CD Pipeline
  • In progress cxserver
    • Images built via deployment pipeline
    • Namespaces created for k8s eqiad/codfw

• • In progress citoid
    • Images built via deployment pipeline
    • Deployed
    • Traffic not switched yet

• • changeprop

• •  Done eventgate
    • Image built via pipeline
    • Chart
    • Deployed

• • In progress ORES
    • cf: Dan's comments

General

• Missed talking about last time: CI for Go
  • https://phabricator.wikimedia.org/T209106
  • There is a blubber.yaml now
  • Runs through service-pipline-test and service-pipeline-test-and-publish as of yesterday

• beta code stewardship request
  • Staging discussion -- first canaries then staging
  • Staging supplants some uses of beta, but not all
  • Joe has a thing to ensure services exist in beta during transition
  • Databases hosted on bare metal?
    • No clear cut answer as yet
    • Discussion still ongoing
    • aside: outgoing firewalls in place now as part of transition

TODOs from last time

•  Done TODO thcipriani to make task for continuous deployment, what's missing? a k8s api token on contint1001 to deploy 
  • https://phabricator.wikimedia.org/T217147
• Jenkins attack vectors
  • We worry about escape/security holes
  • Jenkins is not known to be highly secure
  • What about a way to put the logs somewhere else? and make jenkins non-public
    • nginx, for example - they're on disk
    • The real solution here is probably 2 separate instances of Jenkins
  • TODO various attack vectors document to start
  • There's a releases Jenkins - https://releases-jenkins.wikimedia.org
    • Maybe things meant for production use should be built there
    • Hosted on releases1001
    • thcipriani/hashar to work on figuring out what stages happen on which Jenkins
  • +2 meaning deployment is a cultural shift
    • MediaWiki vs MediaWiki/config

• In progress TODO: support documention like the one tyler did for the portal and pipeline/helmfile and deployment
  • https://wikitech.wikimedia.org/wiki/Deployment_pipeline now exists, https://wikitech.wikimedia.org/wiki/Continuous_Delivery has been deleted.

• TODO: Joe & James_F to work on eventual 2019-04-01 email
  • Beware: announces on 04/01 can be considered an April's fool

• In progress TODO: improve feedback from pipeline -- link to actual failing job, show images, and tags as applicable
  •  Done create task: Pipeline feedback https://phabricator.wikimedia.org/T177868
  •  Done Gerrit Styling patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/490640
  •  Done Groovy in integration/pipelinelib patch: https://gerrit.wikimedia.org/r/#/c/integration/pipelinelib/+/492225/
  •  Done Bot created for commenting https://wikitech.wikimedia.org/wiki/User:PipelineBot
  • Add credentials PipelineBot credentials to jenkins
  • Patch service-pipline-* jobs to use new integration/pipelinelib code
  • Aim for end of week

RelEng

• Proposals written by Dan could use feedback if anyone has any:
  • The pipeline should provide a way to save artifacts from a stage
  • .pipeline/config.yaml Proposal The Latest™ (top level task still ORES one "The continuous release pipeline should support more than one service per repo")

• Local charts for dev
  • https://gerrit.wikimedia.org/g/releng/local-charts
  • How to push docker images somewhere for local development
  • Parsoid image for local development or MediaWiki for local dev
  • docker-pkg for localdev images, upload from contint1001
    • 'TODO: Jeena, Brennen, Tyler to sort specific process

• CI future working group: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/CI_Futures_WG
  • Blog on Phabricator later today
  • Has just started, a prelim list of requirement and candiadates on above page

• Would it be feasible to use spare servers for CI in the future? Even if they're only available for a short period?
  • Alex: There are resources for CI coming up, quite a large amount of money

Serviceops

• citoid deployed on staging and production
  • Still working out the kinks (e.g. metrics)
  • Need to switch traffic to it and deprecated the old method.
• eventgate done \o/

As Always

• Release Pipeline Workboard
• Meeting notes