Wikimedia Release Engineering Team/Checkin archive/20191023
2019-10-23
editVacations/Important dates
edit- October 8: Željko holiday (Independence Day)
- October 11: Željko conference (https://2019.webcampzg.org/ )
- October 14 (Indigenous Peoples' Day): US Staff
- October 24–27: James at WikidataCon 2019 in Berlin
- November 1 (All Saints' Day): Željko holiday
- November 8–10: WikiCon North America (James)
- November 11 (Veterans' Day): US Staff
- November 12–15: TechConf19
- November 16–20: Offsite
- November 21-30: Jeena
- November 25-29: Dan
- November 28–29 (Thanksgiving): US Staff
- December 1-2: Jeena
- December 6: Lars (Finnish Independence Day)
- December 25–31 (Christmas): US Staff
- December 25–26: Lars, Željko, Christmas
- 1 January (New Year's Day): US Staff, Lars, Željko
- 20 January (Martin Luther King Jr. Day): US Staff
Rotating positions
editTrain
edit- Maniphest query for deployment blocker tasks: https://phabricator.wikimedia.org/maniphest/query/s3KW8bpsXhYF/#R
- 14 Oct - wmf.2 - Jeena + Lars
- 21 Oct - wmf.3 - Lars + Brennen
- 28 Oct - wmf.4 - Brennen + Mukunda
- 4 Nov - wmf.5 - Mukunda + Antoine
- 11 Nov - wmf.6 - No Train (Tech conf)
- 18 Nov - wmf.7 - No Train (Off site)
- 25 Nov - wmf.8 - Antoine + Jeena (Thanksgiving)
- 2 Dec - wmf.9 - TBD
- 9 Dec - wmf.10 - TBD
- 16 Dec - wmf.11 - TBD
- 23 Dec - wmf.12 - No Train or any other deploys (Winterval)
- 30 Dec - wmf.13 - No Train or any other deploys (Still Winterval)
- 6 Jan - wmf.14 - TBD
- 13 Jan - wmf.15 - TBD
SoS
edit- Aug 14 onward: Zeljko 🎸 🎷 \o/
Team Business
editTimespent spreadsheet
edit- For the avoidance of doubt: fill out the sheet week number for the previous week
- spreadsheet for week 42 (2019-10-14):https://docs.google.com/spreadsheets/d/1YS_2eUQWUOnygshMfxgGPu93GKGCO5t00yjUREQ5B-I/edit#gid=442607278
- reminder: your time spent doesn't need to total to 100%
Clinic Duty Proposal
editNovember Offsite/Summit
edit- TechConf19: November 12–15
- Offsite: November 16–20
- Arrive the 15th
- 16th is our fun day/outing
- 17–20 are working days
- 21st fly home
- https://etherpad.wikimedia.org/p/EngProdTeam-Offsite-2019-11-planning
- Dietary restrictions due TODAY!
- New proposal on etherpad: how does EngProd want to use betterworks
- Outing day! Vote please
Book club
edit- https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Book_club
- Good discussion! https://etherpad.wikimedia.org/p/releng-book-club
Monthly reflection on accomplishments - October '19 edition
edit- https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments
- Add as you have them!
October
edit- Dev images are now automatically created as part of postmerge via the pipeline for:
- Parsoid
- Soon: RestBASE
- Selenium documentation updated https://www.mediawiki.org/wiki/Selenium/Node.js
- Quibble 0.0.36 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092658.html
- Quibble 0.0.37 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092660.html
- Quibble 0.0.38 & 0.0.39 released for mediawiki/tools/api-testing
- Introducing Phatality - Streamlined error reporting from Kibana to Phabricator https://phabricator.wikimedia.org/phame/post/view/177/introducing_phatality/
- HHVM removed from CI and MediaWiki.
- Gerrit is on gerrit1001 now
- … and so is most of the code review. ;-)
Incoming/Needs attention
editScrum of Scrums for next week
editInternal team notes, not for pasting into the SoS notes
editIncoming from last week (aka this morning)
editOutgoing next week
edit- Blocked:
- Site Reliability Engineering: logstash-beta.wmflabs.org does not receive any mediawiki events https://phabricator.wikimedia.org/T233134
- Updates:
- v3 of CI arch document for new CI wants review and feedback https://www.mediawiki.org/wiki/User:LarsWirzenius/NewCI
- Gerrit is now on a newer (larger) server
- There was some problem with rollout (patches looking unmerged that were actually merged)
- Resolved now
- All gory details on https://phabricator.wikimedia.org/T236114
- Train Health
- Last week: 1.35.0-wmf.2 - phab:T233850
- This week: 1.35.0-wmf.3 - phab:T233851
- Next week: 1.35.0-wmf.4 - phab:T233852
Callouts
editTrain status and happenings
edit
Standup!
edit- Make sure your tasks are correct on our kanban board!
Antoine
editLast week
edit- Best thing
- Paired git fixage with Tyler for hmm lot of hours. Pair programming bash over Google Meet is a great experience.
- For Wikibase.git : extracted Selenium to a standalone jobs ( https://phabricator.wikimedia.org/T232759 )
- Quibble 0.0.39 for mediawiki/tools/api-testing (very cool project)
- Worst thing
- What I planned vs did
- Switch integration/config to use jenkins-job-builder from pypi instead of our git repo
- PostBuildScript jenkins plugin upgrade prepared
- Nerd snipped on some weird logstash index fields limit https://phabricator.wikimedia.org/T234564
- Unplannned
- Gerrit post migration magic (thank you Tyler!) (thcipriani: thank you!)
This week
edit- What I plan to do
- TechConf preparation with Kosta / WMDE-Fisch
- Blockers
- WMCS blocked: they need CI for a golang projects https://phabricator.wikimedia.org/T236203 (dduvall: we have a docker-pkg-built golang base image; see blubber's .pipeline/blubber.yaml)
- php-xdebug needs upgrade (slows down phpunit coverage report)
Other
editBrennen
editLast week
edit- Best thing
- Pairing in general, messing with sshecret.
- Worst thing
- Moments of panic about logs prior to deploy.
- What I planned vs did
- Done Make sure local-charts install script works on buster
- Done Further TechConf input, CI session planning
- In progress Train pairing
- Unplannned
- Some time investigating broken mediawiki/core dev images
- Some time investigating toollabs to restart dockerregistry
This week
edit- What I plan to do
- Train
- Sort out all hands travel
- Dev environment hackery
- Check out k3sup?
- Think about TechConf local dev sessions / discuss
- Improve sshecret pull req: https://github.com/thcipriani/sshecret/pull/2
- Blockers
Other
edit
Dan
editLast week
edit- Best thing
- Diagramming Seakeeper subsystems
- Worst thing
- Fixing janky Jenkins reports
- starting in August, 30-day reports should be accurate
- Fixing janky Jenkins reports
- What I planned vs did
- Seakeeper proposal, fleshed out descriptions of CI subsystems for proposed k8s namespaces
- Unplannned
- Fixed problems with Jenkins reports at https://people.wikimedia.org/~dduvall/jenkins
- Build retention differences made for inaccurate 30-day reports. Working around this by aggregating from past build archives.
- Fixed problems with Jenkins reports at https://people.wikimedia.org/~dduvall/jenkins
This week
edit- What I plan to do
- Seakeeper proposal, gathering capacity/resource utilization stats
- Blockers
Other
edit
Greg
editLast week
edit- Best thing
- The TechConf program committee is great to work with, even when we are stressed
- Worst thing
- Time constraint on the Tuning Session slide creation (which had knock off effects of figuring out what the hell these OKRs and metrics really are)
- Effing moving
- What I planned vs did
- techconf
- tuning session
- Unplannned
This week
edit- What I plan to do
- tech conf schedule creation (well, version 2.0 of the schedule) and post it
- engprod logistics setup
- Blockers
- time
Other
edit- TechConf office hours happening next week on IRC, email going out shortly announcing them
- Desire/stretch: create a homepage for Engineering Producitivity somewhere :)
James
editLast week
edit- Best thing
- Made progress discussing with Timo way forward for static configuration work; next step is autogenerating dblists from YAML.
- Worst thing
- Real jetlag for the first time in a few years. Meh.
- What I planned vs did
- Done OOUI migration to node 10 / PHP72
- Not done Fundraising Tech CRM image to PHP70 (could do with input from Antoine) https://gerrit.wikimedia.org/r/c/integration/config/+/540683 for https://phabricator.wikimedia.org/T230446
- Partially done More static configuration work, a bit delayed.
- Done More TechConf thinking/planning.
- Done UBNs as they arise.
- Unplannned
- Renamed "unversioned" PHP (PHP70) images to be explicit
- Antoine: THANK YOU
- Helping Analytics put a "stats for this site" link in the site footer T235803
- Renamed "unversioned" PHP (PHP70) images to be explicit
This week
edit- What I plan to do
- WikiDataCon from tomorrow onwards; back in the US next Wednesday.
- Whatever breaks.
- YAML/static builds of dblists.
- Blockers
- —
Other
edit- —
Jean-Rene
editLast week
edit- Best thing
- Worst thing
- What I planned vs did
- Unplannned
This week
edit- What I plan to do
- Blockers
Other
edit
Jeena
editLast week
edit- Best thing
- finished restrouter chart...hopefully \o/ \O/ \:D/
- Worst thing
- spent time trying to debug Visual Editor not working, unexpectedly there was an error in the browser console
- What I planned vs did
- Not done timespent spreadsheet
- Not done work with zeljko on getting local-charts running - Željko: feel free to pospond until you're done with train
- In progress look at seakeeper proposal
- Done train
- Done update restrouter deployment-charts config for dev purposes
- Unplannned
- debug broken mw core image
This week
edit- What I plan to do
- tech conf session planning
- changes to mw core image publish pipeline to log more and fail if prereq script errors
- re-open patchsets to build restbase dev image
- timespent spreadsheet
- work with zeljko on getting local-charts running
- ...
- Blockers
Other
edit- tiny sunflower
Lars
editLast week
edit- Best thing
- Going to TechConf
- Worst thing
- Not going to a mountain in TN
- What I planned vs did
- Done Help Jeena with train, a little, as backup.
- Done Seek feedback on v3 of CI arch doc.
- Done Update engprod on PGP signing for offsite.
- Not done Set up Argo in a K8s cluster somewhere.
- Needs uninterrupted time, and I've been easily distractable. Also train.
- Unplannned
- Was invited to TechConf, and did much arranging of travel.
- More review of Dan's Seekeeper proposal.
- Wrote up book club discussion on wiki.
- Failed to deploy train, didn't get to group0 even, but much excitement was had by everyone.
- Wrote a wrapper around rsync, in case it's useful some day.
- Registered for All Hands, started arranging travel.
This week
edit- What I plan to do
- Train, 1.35.0-wmf.3
- Skim Go book, write a small program from scratch: fetch a web page, diff against previously fetched version.
- Attempt to set up study groups, or possibly SIGs, for Go and Rust.
- Blockers
Other
edit- The Darkness is here until about March or April, expect SADness
- performance has TT; releng has BB, DD, and MM; qte has no doubled initials
- :-)
- EngProd has GG :P
- My "legal" name is Tyler Henry Turley-Cipriani, so TTish
Mukunda
editLast week
edit- Best thing
- Phab1001 is up and running
- Worst thing
- Didn't get around to testing the "Ferret" Fulltext search engine in phab (yet)
- What I planned vs did
- Mostly Done Finish getting branch.py ready for the wmf production branching (WIP: https://gerrit.wikimedia.org/r/#/c/mediawiki/tools/release/+/543248/ )
- Done Try to build some tests for branch.py
- Done More work on phabricator high-availability & disaster recovery planning.
- phab1001 should be nearly ready to become the primary server now.
- Not done Test out phabricator's built-in search engine with an eye toward dropping elasticsearch if the integrated search performs as well.
- Unplannned
This week
edit- What I plan to do
- Test out phabricator's built-in search engine
- Drum up interest in patch-triage (email wikitech-l, not sure what else to do)
- Get branch.py ready for full automation in releases-jenkins
- Deploy phatality (after https://gerrit.wikimedia.org/r/#/c/545593/ merges)
- Deploy phabricator
- Blockers
- code review of https://gerrit.wikimedia.org/r/#/c/545593/
Other
edit
Tyler
editLast week
edit- Best thing
- Seeing the train pairing thing work :)
- pairing with Antoine on Gerrit migration fallout
- Worst thing
- Gerrit migration fallout
- What I planned vs did
- Done connect with SRE re New CI (again)
- Done Reach out to folks about being on Sr. SWE interview loop
- Done announce dec code freeze
- Done? tuning session prep
- Done Gerrit -> gerrit1001
- Unplannned
- Gerrit migration fallout
- Updates to deployment-calendar scheduler -- awight noticed some issues
This week
edit- What I plan to do
- Resumé review
- Work out clinic duty rotation
- Next bookclub email
- Gerrit/JVM tuning to make use of Moar Memory™
- Blockers
- Time
- Dog
- Blazey at conference
Other
edit- Picking up Blazey at airport this evening, may cut out a bit early
Zeljko
editLast week
edit- Best thing
- Email from Safeguard: "Please note: all documentation must be password protected from this point using the following (password)"
- Worst thing
- T236282 `npm run selenium` fails with `SyntaxError: Unexpected identifier`
- What I planned vs did
- In progress T234610 ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73
- In progress T199113 All repositories with Selenium tests should use wdio-mediawiki
- Unplannned
- Safeguard
- T235795 Create test for GrowthExperiments to check for ability to disable homepage
- T233820 mediawiki/core Selenium tests fail when targeting mediawiki/vagrant VM
- T236282 `npm run selenium` fails with `SyntaxError: Unexpected identifier`
This week
edit- What I plan to do
- T234610 ERROR webdriver: Request failed due to Error: session not created: Chrome version must be between 70 and 73
- T199113 All repositories with Selenium tests should use wdio-mediawiki
- Blockers
Other
edit- Saw Rambo: Last Blood. Not recommended.