Wikimedia Release Engineering Team/Checkin archive/2024-06-26

2024-06-26

• Last time

🏆 Wins/winterrogation

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments

June 2024

• "Wikimedia Deployment Scheduler" tool is now linked from Gerrit changes and helps add things to backport windows. https://schedule-deployment.toolforge.org/ More at https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/VT44HXYMEYUNDVIGGLII7XZZTNCXA52S/
• Improved httpbb failed header check output: https://gerrit.wikimedia.org/r/c/operations/software/httpbb/+/1037156
• scap sync-world: k8s: image build errors are now fatal. https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/329
• Reggie manifest cleaner deployed (results in one week)
• Restrict some scap subcommands to deploy servers only https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/331

 (plus followup from Jaime)

• Yesterdays phab deploy mitigating some phab vandalism vectors
• GitLab version 17 upgrades—deprecated endpoints in our tools https://phabricator.wikimedia.org/T365675
• Fixed https://phabricator.wikimedia.org/T364309: deployment: fix-staging-perms fails to finish [ set umask on more scap commands ]
• Fixed https://phabricator.wikimedia.org/T366217: unable to create revert commit from scap
• Fixed https://phabricator.wikimedia.org/T366844: Don't just append names with "and" (scap)
• Fixed https://phabricator.wikimedia.org/T366856: `UNIQUE constraint failed: blob.name` during manifest upload to Reggie
• https://gerrit.wikimedia.org/r/c/operations/puppet/+/1041746: logstash_checker.py: Add --time option merged but soon to be obsolete
• https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/360 kubernetes: Always use replicas:1 in traindev
• Deployed buildkitd 0.14.1 to staging and prod gitlab-cloud-runners, and trusted runners (https://phabricator.wikimedia.org/T367352) \o/
• https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/358 Move logstash checker code into scap, and behavior changes. \o/
  • Merged and deployed with scap 4.89.0.
  • Closed https://phabricator.wikimedia.org/T159991 (If aborting a scap due to test canary error rate, output some errors for reference)
  • Closed https://phabricator.wikimedia.org/T183999 (Scap canary has a shifting baseline)
  • Closed https://phabricator.wikimedia.org/T367131 (Did retrying canary checks do anything?)
• Phab
  • AFAIK done with Phab custom downstream TechDebt cleanup, no more random breakage (sorry brennen)
  • Antivandalism DB query code way way more performant - https://phabricator.wikimedia.org/T366811 \o/
  • Misc upstream Phorge work, e.g. showing "this is a dup" in comment field: https://we.phorge.it/F2231629 and "Allow collapsing/expanding workboard column content by clicking its header" for mobile in https://we.phorge.it/D25672, awaiting review
  • Wrote PoC/WIP patch to ignore milestones in a Herald rule "none" condition in https://phabricator.wikimedia.org/T144041 (requested by numerous WMF teams over the last years - currently you need to update your team's backlog funneling Herald rule every time you create a new milestone project)
• gitlab-settings/configure-projects stuff runs on a timer on GitLab prod box \o/ (laptop no longer in prod)
• Actually turned on Phorge integration on GitLab \o/!!
• Can deploy a mediawiki with patches to kubernetes from patchdemo!!!
• Feature flagged catalyst on patchdemo
• Working on BDD tests for Catalyst
• PatchDemo bootstrap scripts in k8s
• Buster deprecations in WMCS
• Blubber experimental/native-llb branch seems stable enough to merge
  • Thinking a 1.0.0 release is in order
  • Docker-ui package integration work
• Gerrit on 3.10 as of today

📅 Vacations/Important dates

https://office.wikimedia.org/wiki/HR_Corner/Holiday_List#2024

https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Time_off (page needs updating for Dayforce)

• Wed Jun 5- Sun 9: Brennen PTO
• Mon Jun 10–Fri 14: Tyler
• Thu Jun 13-Sat 15: Andre less available (DevConf.cz)
• Mon Jun 17-Fri 21: Bryan PTO
• Wed Jun 19: Juneteenth (US staff with reqs)
• Fri Jun 28: Ahmon PTO
• Fri Jun 28: Jaime
• July 1-9: Jaime
• July 4-5: US staff holiday
• July 4-5: Andre (festival; CZ Holiday)
• July 12–15: Ahmon
• July 12-22: Dan
• July 25-31: Ahmon
• Aug 05-09: Dan (I haven't told Tyler yet, shhh)
• Fri 09 Aug – Global holiday: International Day of the World’s Indigenous Peoples
• Sun 25 Aug - 03 Sep: Brennen
• Sept 19-20, 23: Bryan Riot Fest in Chicago!
• Sept??? Jeena

🔥🚂 Train

https://versions.toolforge.org/

https://train-blockers.toolforge.org/

https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar

Rotation

• 13 May (13-17) - 1.43.0-wmf.5 – Antoine + Andre (Jeena as backup)
• 20 May (20-24) - 1.43.0-wmf.6 – Andre + Antoine (you can do it Andre!) (Antoine out Mon, Ahmon out Fri)
• 27 May (27-31) - 1.43.0-wmf.7 – Ahmon + Andre (Memorial day Monday)
• 03 Jun (03-07) - 1.43.0-wmf.8 – Dduvall + Ahmon (Brennen out)
• 10 Jun (10-14) - 1.43.0-wmf.9 – Brennen + Ahmon
• 17 Jun (17-21) – 1.43.0-wmf.10 – Jaime + Brennen
• 24 Jun (24-28) – 1.43.0-wmf.11 – Jeena + Jaime
• 01 Jul (02–04) – 1.43.0-wmf.12 – Antoine + Jeena
• 08 Jul (08-12) – 1.43.0-wmf.13 – Andre + Antoine (Ahmon + Dan out Friday)
• 15 Jul (15-19) – 1.43.0-wmf.14 – Ahmon + Andre (Dan Out)
• 22 Jul (22-26) – 1.43.0-wmf.15 – Dan + Ahmon (Ahmon out Thurs)
• 29 Jul (19-Aug 02) – 1.43.0-wmf.16 – Brennen + Dan (Ahmon out)

Team Discussions

• Noticed this in a weekly update: GitLab/HDFS sync—we may have Opinions™ https://docs.google.com/document/d/1gytt1rzO5wO1IWmmrGKdfuwSlVvFyQ9pFK87wePUYFE/edit

• Friday deployment and the train: how are things working? Any reason to tweak anything there?
  • Context: I often see post-train emergency deploys go out on Fridays—anything that needs to be codified into our process there?

Let's do some inbox triage: https://phabricator.wikimedia.org/maniphest/query/7vRDrcVnt8OI/#R

🌻 Open source/Upstream contributions

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Upstream