Wikimedia Product/Wikimedia Product Infrastructure team/Service deployment process
In a nutshell: when a patch in Gerrit is merged to the master branch of a service repo, a post-merge CI job is run to build a Docker image, according to the definition in the project's Blubberfile, from the updated master branch. Deploying the service simply means updating the service configurations on deployment.eqiad.wmnet and any Beta Cluster instances with the resulting image tag.
Helm charts defining the production service configurations are stored in the operations/deployment-charts repo. To deploy a new version to production, a patch must be created and merged updating the per-cluster service definitions under
helmfile.d/ with the Docker image tag created in the post-merge
service-pipeline-test-and-publish job. Three deployment clusters are defined:
codfw (with the latter two corresponding to Wikimedia's two core server clusters). An example patch updating each with a new Docker image tag for deployment is here. These patches can be self-merged by the deployer at deploy time. After merging, allow a minute or two for the repo on the production server to be synced with the master branch.
The next step is to sync the change to the Kubernetes cluster. On our production deployment servers (accessible as
deployment.eqiad.wmnet) the deployment-charts repo lives at
/srv/deployment-charts. Navigate to the service's helmfile directory for the staging cluster, examine a diff of the changes, and if the diff contains no surprises, sync the change:
$ ssh deployment.eqiad.wmnet $ cd /srv/deployment-charts/helmfile.d/services/staging/<service>/ $ source .hfenv; helmfile diff $ source .hfenv; helmfile apply
If everything looks OK, repeat the process in
Deploying to the Beta Cluster involves a similar procedure. Log into Horizon, navigate to the deployment-prep project, select Compute > Instances from the left-hand nav menu, and click on the name of the instance to which you intend to deploy. In the instance view, click the Puppet Configuration tab and observe the Hiera Config section of the resulting screen. In the configuration shown, you should see a
profile::docker::runner::service_defs setting containing a familiar-looking service-runner service configuration; find the
version key under
profile::docker::runner::service_defs and update it with your new image tag. The instance will pick up the updated configuration during the next Puppet run.
- Wikimedia Apps/Team/RESTBase services for apps/Deployment process — historical deployment process