Wikimedia Services/deploying
WORK IN PROGRESS. Shannon Bailey WMF sbailey
How to deploy an updated and recently merged service like proton or wikifeeds to production
This guide provides recommendations and the steps to accomplish a deployment of a typical update to a service. The chromium-render (proton) wiki page to PDF print service is used as the primary example throughout this guide.
This guide presumes you have a locally tested, reviewed and merged patch on gerrit for the service you are working on which is based on the current master branch. It also presumes that you have been given deployment access or are working with another engineer who has deployment access. It also presumes that you have contacted a colleague who is available to review your deployment patch and +2 it soon after you have completed the git review process, to allow you to proceed with the steps to be carried out on the deployment server. If you do not have deployment server access, your colleague can carry out those steps on your behalf if, but be sure to arrange that with them and it is best if they also reviewed the changes to the service you are making, allowing you both to test it while in staging prior to it going live in the public access servers.
Before attempting to deploy your service, testing it from the command line if possible should be figured out and performed locally as part of development and test and hopefully easily converted to run on a staging server with minor tweaks to the curl command. For example, to test the proton PDF renderer locally the following curl command runs a local request with the server running on localhost (TODO: INSERT CURL COMMAND HERE FOR LOCAL EXECUTION). On the staging server it requires a different curl URL. Running a curl request with -v (verbose output) should produce a report and if it gets a http 200 response it is working. To determine the port the service will be hosted on, on the staging server, use the URL below to determine the proper service port. The proton service currently is assigned port 4030 and wikifeeds service is assigned port 4101. Look this up and keep it ready so you can form your curl request when needed.
This example URL makes a request to the proton service on the staging server from the deployment server’s terminal console.
curl https://staging.svc.eqiad.wmnet:4030/en.wikipedia.org/v1/pdf/Foo/a4/desktop -v
Here is another example URL to make a request to the wikifeeds service from the deployment server.
curl https://staging.svc.eqiad.wmnet:4101/tr.wikipedia.org/v1/aggregated/onthisday/all/01/05 -v
With these preparations made, let’s get to deploying!
The first step to deploying an updated service is to launch a local terminal session and change to, or create a ‘work’ directory to hold the deployment-charts repository code in.
% cd work
work % ls
deployment-charts
If there is already a work/deployment-charts directory, change to that directory
work % cd deployment-charts
then update it to current with a
deployment-charts % git pull
followed by a
deployment-charts % git remote update
Make sure you have checked out master and are not on a previous branch. You should see the following results which includes other changes made to the repo since it was last updated.
otherwise go to https://gerrit.wikimedia.org/g/operations/deployment-charts and get then git clone the URL specified there from within your % work directory. This page is also where you will see the final results of your committed changes to this repository once they have updated in the system which can take a few minutes.
work % git clone https://gerrit.wikimedia.org/r/operations/deployment-charts
I recommend branching at this time to avoid working on master directly.
work % git branch updateProton
work % git checkout updateProton
Switched to branch 'updateProton'
Now that you have a current deployment-charts repo under an appropriate branch, you need to navigate to the appropriate directory for the service you are deploying. In this example we are updating the proton service, but substitute your service name after the …/services/ to be in the correct directory for your service
work % cd deployment-charts/helmfile.d/services/proton
Check the directory contents and confirm you are in the right directory and see the values.yaml file
Now that you are in the right directory, use an editor such as vi to edit values.yaml
The current main_app: version: tag will be visible and should be checked while updating it to the new value to double check this is the current version in the repo
Now we need the latest tag for the service commit that you recently merged. This is found in the docker registry.
https://docker-registry.wikimedia.org/
You will need to search this long web page for the service you are updating. For this example the chromium-render (proton) service is of interest.
Clicking on the link should bring you to the the Tags: info for your service
Now scroll down to the newest tagged build we need to copy and paste into values.yaml version:
You should see the current value in the editor being the second to last entry
2024-01-29-184230-production
If it is farther up the list, it may be that another deployment has occurred and your patch is not the latest one immediately after the currently deployed one, so double check that. It should be the second to last tagged release like in this case. If it is not, your patch may need to be rebased on the current master to ensure it includes all changes that were made while you were working on it, while another engineer had made some changes and deployed their changes after you began working on the service.
The last tagged item should be like it is in this example.
2024-02-05-181957-production
If this all checks out replace the version: with this new tag using the editor as shown here and save the file and exit the editor back to the console.
It is common practice to do a git diff to ensure that the changes you intend are present and no other unintended change is present.
proton % git diff
If this looks good it is time to commit these changes and publish them, so stage them and commit the change.
proton % git add -u
In the commit message, the convention is to include the service name followed by ‘bump to latest version’, and it this case was ‘proton: bump to latest version’
proton % git commit
You should see a message like this in the commit response after the commit completes successfully.
[updateProton 77d0c403] proton: bump to latest version
1 file changed, 1 insertion(+), 1 deletion(-)
Now that the commit has succeeded, it is time to push the committed patch to gerrit for review by a colleague.
proton % git review
On success, copy the URL from the review response and open it in your browser. Add the name of a person who is ready to review your patch as a reviewer and wait for the patch to be +2 and jenkins-bot to merge the patch.
Congratulations, you are close to getting the service running on a staging server so you, or your colleague with deployment server access, can test it and then deploy to production.
TO BE CONTINUED THROUGH STAGING DEPLOYMENT AND TESTING AND PRODUCTION DEPLOYMENT