Wikimedia Release Engineering Team/Checkin archive/2024-08-14

2024-08-14

• Last time

🏆 Wins/winterrogation

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments

August 2024

• Fixed issue with deployment-deploy04 free space. Added a 40GB volume and copied /srv to it.
• Buildkit 0.15.1 release deployed
• Helped data-engineering Airflow DAGs with their Gitlab CI.
• Rewrote remainder of make-container-image stuff in Python: https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/99 \o/
  • Scap invokes this repo during deployment
  • Current status: Create a php7.4 image + debugging packages
  • Future: php7.4 + php8.1
  • Single version images: there's a change in mw-config to override wikiversions.json
• Updated train-dev to use debian:11 base image.
• Kicked off nomination process to reboot the Toolforge standards committee (https://phabricator.wikimedia.org/T370474)
• Moved a tool to toolforge build service
  • https://extloc.toolforge.org/
• Fixed links in patchdemo for catalyst wikis
• merged persistence for k8s patchdemo
• Added read-only flag for patchdemo
• Fixed a Phab code bug not checking user permissions creating a form
• Merged more Phorge upstream stuff to get bugfixes + features once we pull, e.g. logging errors for broken Herald rules. See some stuff as deps: https://phabricator.wikimedia.org/T370266 (when downstream tasks exist)
• Played with checking for active Phab accounts linked to locked WMF SUL accounts (TODO: other way round)
• Started working on a Kubernetes cluster for deployment-prep using OpenTofu and Magnum as provisioning tools. Lots of things to figure out still, but a proof of concept cluster was provisioned, destroyed, and provisioned again. https://gitlab.wikimedia.org/bd808/deployment-prep-opentofu + deploymentpreps3
• Scap deploy with rewrite of build-image script
• Merged catalyst/patchdemo environment redirects

📅 Vacations/Important dates

https://office.wikimedia.org/wiki/HR_Corner/Holiday_List#2024

https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Time_off

• Aug 02: Bryan
• Aug 05-08: Dan
• Fri 09 Aug – Global holiday: International Day of the World’s Indigenous Peoples
• Aug 12: Dan
• Mon 12 Aug-16: Ahmon out
• Mon 12 Aug - Fri 23 Aug: Antoine
• Aug 16: Bryan
• Aug 23: Bryan
• Aug 23: Jaime
• Sat 24 Aug - 03 Sep: Brennen 🔥
• Aug 30: Bryan
• Sept 02: US Labor day (WMF US holiday)
• Sept 06: Bryan
• Sept 13: Bryan
• Sept 18-19: Brennen Winfield
• Sept 19-20, 23: Bryan Riot Fest in Chicago!
• Sept23-27: Andre
• Sept24: Dancy
• Sept 27: Bryan
• Sept 11-30 Jeena
• Sept 27: Dan
• Oct 03-06: WikiCon North America (Indianapolis)
• Oct 6: Dancy
• Oct 1-11 Jeena
• Oct 14: Indigenous Peoples' Day (also Columbus Day) US Staff w/reqs

🔥🚂 Train

https://versions.toolforge.org/

https://train-blockers.toolforge.org/

https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar

Rotation

• 05 Aug (05-09) – 1.43.0-wmf.17 – Jaime + Brennen (Dan out, Global holiday Friday)
• 12 Aug (12–16) – 1.43.0-wmf.18 – Jeena + Jaime (Ahmon out, Antoine out)
• 19 Aug (19–23) – 1.43.0-wmf.19 – Andre + Jeena (Antoine out)
• 26 Aug (26–30) – 1.43.0-wmf.20 – Antoine + Andre (Brennen out)
• 02 Sep (02–06) – 1.43.0-wmf.21 – Ahmon + Antoine (US holiday Monday, Brennen out Tues)
• 09 Sep (09–13) – 1.43.0-wmf.22 – Dan + Ahmon
• 16 Sep (16–20) – 1.43.0-wmf.23 – Jaime + Dan (Brennen out)
• 23 Sep (23–27) – 1.43.0-wmf.24 – Brennen + Jaime (andre out)

Brief demo of all the grafana data in digital ocean

• Background of this is trying to do horizontal pod autoscaling, but gitlab scaling has this
• Kubernetes needed access to information collected in prometheus
• Never done anything with grafana, gitlab job info may be in prometheus
• How to get here:

 kubectl get ns
 kubectl --namespace monitoring get po
 kubectl --namespace monitoring describe po kube-prometheus-stack-grafana-cd65fd5f5-wtpfz 
 kubectl --namespace monitoring exec kube-prometheus-stack-grafana-cd65fd5f5-wtpfz -c grafana -- env
 kubectl --namespace monitoring port-forward service/kube-prometheus-stack-grafana 8081:80

Some things I looked at trying to get to it.

The config map:

 kubectl --namespace monitoring describe configmaps kube-prometheus-stack-grafana

Seeeeccreetts (secrets):

 kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana
 kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath='{.data}'

logs:

 kubectl --namespace monitoring logs kube-prometheus-stack-grafana-cd65fd5f5-wtpfz
 kubectl --namespace monitoring logs kube-prometheus-stack-grafana-cd65fd5f5-wtpfz -c grafana

I managed to find the secret in the env—it was not the secret from the secret:

 kubectl --namespace monitoring exec kube-prometheus-stack-grafana-cd65fd5f5-wtpfz -c grafana -- env

Question:

• What does the exporter export to promethus?
• Pod annotation aggregation in grafana?
• Jobs things in prometheus that we collect?
• https://docs.gitlab.com/runner/monitoring/#available-metrics

Looming essential project roundup

• Zuul upgrade
• Phabricator hosts on PHP 8.x
• Phabricator on k8s???
• Onboarding/offboarding
• Deployment-prep beyond we6.2 (group -1/catalyst)
• Attestation???
  • docker-pkg provenance
  • make-container attestation
  • Hooking this to deploy tools
• helm deploy continuous deployment

🌻 Open source/Upstream contributions

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Upstream