Wikimedia Release Engineering Team/Checkin archive/20191002
2019-10-02
editVacations/Important dates
edit- September 23–October 4: JR Vacation
- October 8: Željko holiday (Independence Day)
- October 11: Željko conference (https://2019.webcampzg.org/ )
- October 14 (Indigenous Peoples' Day): US Staff
- November 11 (Veterans' Day): US Staff
- November 12–15: TechConf19
- November 16–20: Offsite
- November 28–29 (Thanksgiving): US Staff
- December 6: Lars (Finnish Independence Day)
- December 25–31 (Christmas): US Staff
- December 25–26: Lars, Christmas
- 1 January (New Year's Day): US Staff, Lars
- 20 January (Martin Luther King Jr. Day): US Staff
Rotating positions
editTrain
edit- Maniphest query for deployment blocker tasks: https://phabricator.wikimedia.org/maniphest/query/s3KW8bpsXhYF/#R
- June 24 - wmf.11 - Jeena (with Mukunda)
- July 1 - wmf.12 - No train (Fourth of July)
- July 8 - wmf.13 - Jeena
- July 15 - wmf.14 - Lars (with Antoine)
- July 22 - wmf.15 - Lars
- July 29 - wmf.16 - Brennen (with Tyler)
- Aug 5 - wmf.17 - Brennen
- Aug 12 - wmf.18 - No Train (Wikimania)
- Aug 19 - wmf.19 - Zeljko 😱
- Aug 26 - wmf.20 - Zeljko 😭
- Sep 2 - wmf.21 - Antoine
- Sep 9 - wmf.22 - Antoine
- Sep 16 - wmf.23 - Mukunda
- Sep 23 - wmf.24 - Mukunda
- Sep 30 - wmf.25 - Tyler Dan
- 7 Oct - wmf.26 - Tyler Dan
- 17 Oct - 1.35.0-wmf.1 - TBD
- 21 Oct - wmf.2 - TBD
- 28 Oct - wmf.3 - TBD
- 4 Nov - wmf.4 - TBD
- 11 Nov - wmf.5 - TBD (? No train – TechConf ?)
- Who is available from EngProd + CPT + SRE?
- 18 Nov - No train – EngProd offsite
- 25 Nov - wmf.7 - Antoine? (? No train – Thanksgiving ?)
- EU Train week?
- Or just run with a delayed "Thursday" to the next Monday?
SoS
edit- Aug 14 onward: Zeljko
- SoS on 2019-10-09 is at the same time as Tech All Hands
Team Business
editTimespent spreadsheet
edit- For the avoidance of doubt: fill out the sheet week number for the previous week
- spreadsheet for week 39 (9/23): https://docs.google.com/spreadsheets/d/1UEmjIbv6voTF5bOnHdjOltw47AwPDtnmUo0h78wgYDY/edit?pli=1#gid=131825410
November Offsite/Summit
edit- TechConf19: November 12–15
- Offsite: November 16–20
- Arrive the 15th
- 16th is our fun day/outing
- 17–20 are working days
- 21st fly home
- https://etherpad.wikimedia.org/p/EngProdTeam-Offsite-2019-11-planning
Book club
edit- https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Book_club
- 2019-10-15
- Read the blog post: http://blog.codepipes.com/testing/software-testing-antipatterns.html
- Let's discuss! https://etherpad.wikimedia.org/p/releng-book-club
- 2019-10-15
Monthly reflection on accomplishments - September '19 edition
edit- https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Monthly_notable_accomplishments
- Add as you have them!
- Scap 3.12.1-1 released/deployed
- Refactored Zuul layout to use per-branch pipelines
quibble -c
Lets you run arbitrary code against a working MediaWiki install- The phabricator "Report Error Code" form (https://phabricator.wikimedia.org/maniphest/task/edit/form/46/ ) has been updated with separate fields for the stack trace and error code/request id.
- T232608 Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days
- Write cached config to JSON as well as serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533592/ (first step towards a saner config)
- MediaWiki PHP support target modernised from 7.0+ to 7.2+ for 1.34 onwards. https://phabricator.wikimedia.org/T228342
- Quibble 0.0.35 release
- 1.34.0-wmf.24 branch cut was done /mostly/ with branch.py instead of make-wmf-branch.php (some small bugs remain to work out but it's very close)
- Creating accounts was broken on beta cluster since 2019-09-08. It was fixed today (2019-09-25). https://phabricator.wikimedia.org/T232796
- Phatality extension for Kibana deployed to production and used for reporting production errors into Phabricator.
- Train blocker tasks created for 1.35.0-wmf.1-1.35.0-wmf.25
- MediaWiki dev images are now automatically created as part of postmerge via the pipeline
Incoming/Needs attention
edit- Pipeline: https://phabricator.wikimedia.org/T233316#5536374
- CPT secrets: https://phabricator.wikimedia.org/T233092
New Month, New Workboard!
edit- Please move from 201909 -> 201910 (and move up to TODO/close invalidate/whatever liberally)
- 201909 https://phabricator.wikimedia.org/project/view/4253/
- 201910 https://phabricator.wikimedia.org/project/view/4311/
- Let me know in 1:1s if you're stuck/have a question about how to treat a task
Proposed New Rotating Position
edit- Clinic Duty!
- Analogous to:
- Proposed scope
- Triage tasks on #together INBOX/Release-Engineering-Team-TODO INBOX/Release-Engineering-Team-TODO (201XXX) INBOX
- Handle "hey releng"-type requests in #wikimedia-releng (@meatbags for brennen)
- +2 common requests (add x@y.com to ci whitelist)
Scrum of Scrums for next week
editInternal team notes, not for pasting into the SoS notes
editIncoming from last week (aka this morning)
edit- The Growth team is planning deployments for the newcomer tasks projects and we need to know November black out dates
- CPT: Create a way to share a secret between MediaWiki and the testing framework in CI phab:T233092
- Kartotherian folks: phab:T233316
- Not blocked, but nice-to-have for Fundraising tech: php7.0 docker image for CRM tests phab:T230446
- Fundraising emails to English-speaking countries started going out this week - that traffic goes through donate.wikimedia.org (on the main cluster) before it hits payments-wiki (on the fr cluster), so let's nobody break it.
Outgoing next week
edit- Blocked:
- Blocking:
- Updates:
- Train Health
- Last week: 1.34.0-wmf.25 - phab:T220750
- This week: 1.35.0-wmf.1 - phab:T233849
- Next week: 1.35.0-wmf.2 - phab:T233850
- Train Health
Callouts
editTrain status and happenings
editOKRs, Goals, Betterworks, Oh my!
edit- Concerns? Questions no one can answer?
- Additionally (yes, that's right)...Team Goals
Standup!
edit- Make sure your tasks are correct on our kanban board!
Antoine
edit- What I planned vs did last week
- Had Castor instance relocated to have more CPU https://phabricator.wikimedia.org/T232646 \o/
- Extract Selenium to standalone job https://phabricator.wikimedia.org/T232759
- Gerrit and java GC debugging (some bot was still hammering the master) https://phabricator.wikimedia.org/T234328
- Found a slowness bug in php-xdebug from component/php72 ( https://phabricator.wikimedia.org/T234418 )
- What I plan to do this week
- More extraction of Selenium to standalone jobs ( https://phabricator.wikimedia.org/T232759 )
- Teach Quibble to record duration of each stages and report at end of builds (partly done)
- OKR?
- Reviews for Quibble
- Stretch VM are slower than the Jessie ones https://phabricator.wikimedia.org/T232706
- What I'm blocked on
- Status of HHVM in production? Which implies: can we drop HHVM from CI?
- Other?
- Some cloudvirt are unreasonably slow. Still going on
- https://phabricator.wikimedia.org/T225713 "CPU scaling governor fix up"
- https://phabricator.wikimedia.org/T223971 "old cloudvirt are CPU slow"
- Logstash on beta has been broken for months https://phabricator.wikimedia.org/T233134
- Some cloudvirt are unreasonably slow. Still going on
Brennen
edit- What I planned vs did last week
- Done generate MediaWiki + Apache dev image from pipeline (!!!)
- Not done Send Lars my PGP key, after I figure out what's wrong with my PGP key
- Done Read Lars's latest CI document
- Done Final(?) CI WG meeting on Friday
- Done Investigate whether https://phabricator.wikimedia.org/T229280 is really still happening
- In progress Focus on dev environment questions
- Unplanned: Reviewed some pipelinelib patches
- What I plan to do this week
- Dev environment experimentation
- Pair with Jeena on generating some service dev images in pipeline
- T234391 Point deployment-charts/mediawiki-dev at latest dev image published by pipeline
- Put OKRs in Betterworks
- Read for book club
- Figure out what violence Enigmail and Keybase did to my GPG config
- What I'm blocked on
- Other?
Dan
edit- What I planned vs did last week
- In progress Seakeeper K8s Proposal
- [yay!]++
- In progress Seakeeper K8s Proposal
- What I plan to do this week
- What I'm blocked on
- Other?
Greg
edit- What I planned vs did last week
- Done Tech-mgt F2F, meet Grant, new CTO
- Done Leveling...
- What I plan to do this week
- TechConf session drafting
- OKR drafting, aligning
- Goal setting cross EngProd
- What I'm blocked on
- The announcement for EngProd creation :) (just waiting, not actually blocking anything other than nice feelies)
- Other?
James
edit- What I planned vs did last week
- Partially done Vacation all week – I worked on Thursday anyway, because I was bored.
- What I plan to do this week
- Supporting 1.34 release branch cut/etc.
- OKR fun.
- HHVM removal from CI and later MW
- Static configuration progress https://phabricator.wikimedia.org/T223602
- Iterating from CodeReview metrics/KPI proposal to make some for CI
- What I'm blocked on
- Seeking input from deployers/configurators on plans for static configuration changes
- Other?
- —
Jean-Rene
edit- What I planned vs did last week
- vacation
- What I plan to do this week
- vacation
- What I'm blocked on
- Other?
Jeena
edit- What I planned vs did last week
- Done Update parsoid blubberfile to new version
- Done local-charts meeting
- Not done Get parsoid image built in CI. Same for restbase?
- Done Fix some issues with mediawiki chart
- Done Fix some issues with parsoid chart
- In progress bring up issue with deployment-charts about templates with same name
- Done Add ideas to local dev brainstorm
- Done local dev meeting
- Done docker sig meeting?
- What I plan to do this week
- make example of deployment charts template problem for Alexandros
- OKR writing
- local dev meeting & prep
- Work with Brennen to get parsoid image built in CI. Same for restbase? Good question.
- Explore what's needed for new php parsoid service in terms of local-charts/deployment-charts
- Reading for book club
- What I'm blocked on
- Other?
Lars
edit- What I planned vs did last week
- Not done Get feedback on v3 of CI arch doc. Process it.
- Done Read up on Zuul dependent pipelines.
- Done With CI WG, decide on CI engine for future CI. Make plan on what next.
- Not done Update engprod list on PGP keysigning at offsite.
- Not done Read the book club article. Let subconsicous start work on it.
- Done filled in my OKRs in betterworks
- What I plan to do this week
- Seek feedback on v3 of CI arch doc.
- Set up Argo in a K8s cluster somewhere.
- Update engprod list on PGP keysigning at offsite.
- Read the book club article. Let subconsicous start work on it.
- What I'm blocked on
- K8s - could do with some help
- Other?
Mukunda
edit- What I planned vs did last week
- Done Train
- Done Get familiar with betterworks
- Done Deploy phatality to production
- Done Develop some OKRs
- What I plan to do this week
- Branch REL1_34 with Dan
- Work on branch.py
- Phabricator Deploy
- Work on getting phabricator to notify you when attempting to submit a duplicate phatality report.
- What I'm blocked on
- Other?
Tyler
edit- What I planned vs did last week
- Done tech-mgr f2f
- What I plan to do this week
- Goals
- Meetings
- Gerrit1001
- What I'm blocked on
- Other?
Zeljko
edit- What I planned vs did last week
- Done T232598 Update existing Selenium documentation
- Not done T232638 Add accessibility check for Homepage (GrowthExperiments)
- What I plan to do this week
- T213268 Upgrade webdriverio to version 5 in mediawiki/core
- T234002 Make MediaWiki Wdio tests less slow (Sept 2019)
- T199113 All repositories with Selenium tests should use wdio-mediawiki
- T234311 Fix and restore daily browser tests for TwoColConflict
- T232638 Add accessibility check for Homepage
- What I'm blocked on
- T233960 Error: error installing: the server could not find the requested resource
- SoS on 2019-10-09 is at the same time as Tech All Hands
- Other?
- Should I be working on individual goals? apparently not
- October 1 was 7 (seven!) years I've been working for WMF