Wikimedia Release Engineering Team/Checkin archive/2023-07-19


2023-07-19

edit

πŸ† Wins & Gratitude

edit
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments
Jul '23 edition
  • Bunch of GitLab accounts created ~200 accounts
    • Tooling for recurring account syncing
  • Default phabricator.wikimedia.org frontpage has a "I need access or permissions to..." tab panel - https://phabricator.wikimedia.org/W2984
  • Building out a deployment pipeline
  • Good reviews and talented engineers on the team
  • Desiloed
  • Did math!
  • JohnBond upgrading to puppet 7
  • Contint servers upgraded to in-warranty hardware! (Dzahn, John Bond, Jelto)β€”ssds! zuul-merger should be way faster
  • git::clone puppet resource updated! LDAP group sync stuff
  • Great team + smart people + good reviews
  • ^ +1 + Daniel Zahn, Eoghan, Ahmon's review
  • Meeting yesterday with Dzahn, grateful for him reaching out, having a plan for phab migration stuff
  • Mention of doing a git blame on stacktraces. Experiment in scap was https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/773870
  • Phatality digging
  • BuildKitd allowlist is deployed to production
  • Phab backlog cleaning
  • Boldly deleting wikipages!
  • Andre has phab-root, gerrit-root, contint-root
  • Toronto is beautifulΒ :)
  • List of supergroupers (DX)

🌻 Open source/Upstream contributions

edit
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Upstream


😢 Let's keep these empty

edit

Code review

edit

Gerrit Access requests

edit

GitLab Access requests

edit

πŸ“… Vacations/Important dates

edit
https://office.wikimedia.org/wiki/HR_Corner/Holiday_List#2023
https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Time_off

July 2023

edit
  • 03-04 Mon: U.S. July Holiday, US staff with reqs
  • 05 Wed & 06 Thu: Andre CZ public holidays
  • 06 Wed–11 Tue: Dancy out (wedding, not mine)
  • 14 Fri Antoine (Bastille Day, France national holiday)
  • 18 Tue - 21 Fri Jaime
  • 23 Fri Jun–18 Fri Aug: Jeena β†’ Mongolia :D :D
  • 25Tue-28Fri Andre less available (conference)
  • 27 Thr - 28 Fri Dan
  • 28 Fri: Brennen probably out for RockyGrass πŸͺ•
  • 31 Mon Jul – 21 Mon Aug Antoine

August 2023

edit
  • 31 Mon Jul – 21 Mon Aug – Antoine
  • 23 Fri Jun–18 Fri Aug: Jeena β†’ Mongolia :D :D :D
  • 07 Fri: Brennen flying back from NYC
  • 09 Wed: International Day of the World's Indigenous Peoples, US staff with reqs
  • 11 Fri: Brennen out for Folks Fest (?)
  • 7-11 Mon-Fri: Dan out for family vacation
  • 27 Aug–04 Sep: Brennen (πŸ”₯)

September 2023

edit
  • 26 Aug–05 Sep: Brennen (πŸ”₯)
  • 13 Weds–17 Sun: Brennen β†’ KS (approximate)

October 2023

edit

πŸ”₯πŸš‚ Train

edit
https://tools.wmflabs.org/versions/
https://train-blockers.toolforge.org/
https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar


  • 2 Jan - wmf.17 - Dan + Antoine (Jaime out)
  • 9 Jan - wmf.18 - Jeena + Dan (Jaime out)
  • 16 Jan - wmf.19 - Jaime + Jeena
  • 23 Jan - wmf.20 - Brennen + Jaime
  • 30 Jan - wmf.21 - Ahmon + Brennen
  • 6 Feb - wmf.22 - Chad + Ahmon
  • 13 Feb - wmf.23 – Dan + Chad
  • 20 Feb - wmf.24 – Antoine + Dan
  • 27 Feb - wmf.25 – Jaime + Antoine
  • 6 Mar – wmf.26 – Jeena + Jaime
  • 13 Mar – wmf.27 – Brennen + Jeena
  • 20 Mar – wmf.1 – Ahmon + Brennen
  • 27 Mar – wmf.2 – Chad Dan + Ahmon
  • 3 Apr – wmf.3 – Antoine + Dan
  • 10 Apr – wmf.4 – Chad + Antoine
  • 17 Apr – wmf.5 – Jaime + Chad
  • 24 Apr – wmf.6 – Jeena + Jaime
  • 1 May – wmf.7 – Brennen + Jeena
  • 8 May – wmf.8 – Antoine + Brennen (Ahmon out + Antoine Out 8th)
  • 15 May – wmf.9 – Ahmon + Antoine (Dan out + Chad out)
  • 22 May – wmf.10 – Chad + Ahmon (Dan out + Jeena out 26th)
  • 29 May – wmf.11 – Dan + Chad (Memorial Day 29th)
  • 5 Jun – wmf.12 – Jeena + Dan (Brennen out, Jaime out)
  • 12 Jun – wmf.13 – Jaime + Jeena
  • 19 Jun – wmf.15 – Cancelled for offsite
  • 26 Jun – wmf.16 – Brennen + Jaime (Jeena out)
  • 3 Jul – wmf.17 – Antoine + Brennen (3rd + 4th holidays)
  • 10 Jul – wmf.18 – Dan + Antoine (Ahmon out)
  • 17 Jul – wmf.19 – Ahmon+Dan (Brennen out Friday)
  • 24 Jul – wmf.20 – Jaime+Ahmon

Team discussions

edit

Contint Decomm

Move Phab to Phorge

  • Migrate ganetti VM to phorge, send around login information
  • Build a testing planβ€”happy path of workflows
  • Investigate aphlict upgrade

Next sprint!

edit

Phab stuff

edit

Objective: prepare for bullseye upgrade

  • 7.4 incompatibilities
  • Get our repo in-order vs upstream (stack patches better)
  • Stretch: pair on deploys
  • Upgrade dev-tools instance?
  • Via Jaime 1-on-1: develop environment

PipelineLib β†’ GitLab

edit

Objective: Migrate 1 SRE service thing

  • GitLab running out of diskspace https://phabricator.wikimedia.org/T340887
    • storage the for the registryβ€”we have a device we can use for storage space
    • Bigger instances
    • Restarting BuildKitd on a timer
    • Figure out how to reduce the number of jobs running on WMCS runners
      • 1031 is instance-wide
      • Common patterns may not work here since there's no registry
      • WMCS vs K8s runners
        • There are pros and cons
        • Proposal: Disable WMCS runners for a week and track the impact
  • Collect data about job failures/job run time