Wikimedia Release Engineering Team/Checkin archive/20171218

2017-12-18

Vacations/Important dates

https://office.wikimedia.org/wiki/HR_Corner/Holiday_List

How to do it

• December Fridays - Greg is taking off
• December 20-22: Dan - holiday
• December 25 (Monday): Željko - holiday (Christmas Day)
• December 26 (Tuesday): Željko - holiday (St Stephen's Day)
• Dec 25-Jan 1 - End of year/new year holidays
• January 1 (Monday): Željko - holiday (New Year's Day)

Team Business

Rotating positions and absences

Maniphest query for deployment blocker tasks: https://phabricator.wikimedia.org/maniphest/?project=PHID-PROJ-fmcvjrkfvvzz3gxavs3a&statuses=open%28%29&group=none&order=newest#R

Dec 18 and Dec 25

• Train: NO ONE!
• SoS: Chad
• Out:
  • Dec 25-Jan 1 - End of year/new year holidays
  • December 25 (Monday): Željko - holiday (Christmas Day)
  • December 26 (Tuesday): Željko - holiday (St Stephen's Day)
  • Antoine - not sure yet of the exact dates

Jan 1 and Jan 8

• Train: Mukunda
  • wmf.15
  • wmf.16
• SoS: Tyler
• Out:
  • January 1 (Monday): Holiday, everyone

Release Engineering

• Blocking
• Blocked
  • Teams that did not port Selenium tests from Ruby to Node.js (T139740): No team (1), Editing (1), Fundraising Tech (1), Global Collaboration (4), Multimedia (1), Reading Web (3), WMDE Engineering (5)
• Updates
  • REMINDER: No deploys this week and next (18th and 25th). Take a break :)

Last week

• Blocking
  • MW Platform team: https://phabricator.wikimedia.org/T182344 - "Blocker must be a local user or a name that cannot be a local user"
• Updates
  • [MW Train] Reminder! This is your last week of deployments for the year/quarter! No non-emergency deploys starts the week of December 18th.
  • [Meta] Q3 goal planning
  • [Security/Ruby] T180878 Upgrade RuboCop and Rubyzip (Ruby)
    • All done except for Minerva (needs a +2) and mediawiki/debian (unsure if needed).
  • [Security/Jenkins] Upgraded a bunch of Jenkins plugins Monday morning EU time (after a bunch of security releases).
  • [Phabricator] Exploring the use of Selenium tests for search quality regressions
  • [TechDebt] Getting further on the definition of “steward”. We will be talking with Victoria and Toby “soon”.
  • [SSD] One more merge (https://gerrit.wikimedia.org/r/#/c/395570/) and our quarterly goals are complete.
  • [PostMortem] ORES post-mortem completed last week: https://etherpad.wikimedia.org/p/Post-Mortem-T181006

Puppet SWAT

list of patches you want to submit to Puppet SWAT

Logspam \ Last week's train updates

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Roles#Train_Conductor

• a couple things with fixes
• going to spend some time this week tidying it up before the holidays

Other Team Business

Q3 goals! :) :)

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Goals/201718Q2

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Goals/201718Q3

• All Technology goals, our's there now: https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q3

Program 1: webdriver.io

https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Final/Programs/Technology#Program_1:_Availability.2C_performance.2C_and_maintenance

• T182421 Selenium framework improvements
  • Things to do in the next quarter.

Program 3: Tech Debt

https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Final/Programs/Technology#Program_3:_Addressing_technical_debt

Outcome 1

• Objective 2: Identify and find stewards for high-priority/high use code segment orphans - task T174091
• Objective 3: Define and steward a light-weight process for adopting or orphaning/sunsetting products and infrastructure.
  • This is done by the Sunsetting working group

Outcome 2

• Objective 2: Define and implement a process to regularly address technical debt across the Foundation
• Objective 3: Promote and surface important technical debt topics at large gatherings of Wikimedia developers (e.g., DevSummit and Hackathon(s))

Program 6: SSD

https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Final/Programs/Technology#Program_6._Streamlined_service_delivery

• The verify stage of the pipeline
  • https://www.mediawiki.org/wiki/File:Containerized_continuous_delivery_2017_concept.png
  • https://wikitech.wikimedia.org/wiki/Streamlined_Service_Delivery_Design#Step_4:_Isolated_deploy_and_e2e_tests
  • When to make new builds
  • Pushing into a k8s cluster (not staging)
  • In the interim do something with GCE for basis end-to-end tests

• Pre-Dependency on Ops:
  • CI k8s cluster for the Verify pipeline stage

Program 1: Outcome 1: Objective 1: Scap (Tech Debt Sprint FY201718-Q2) =

workboard

• SCAP doesn't actually deploy tags or non-default branches. This should be resolved in https://phabricator.wikimedia.org/D918

Program 1: Outcome 5: Objective 1: Maintain existing shared Continuous Integration infrastructure =

Goal: A generalized POC for a docker-based CI.

https://phabricator.wikimedia.org/project/view/3008/ (shipyard workboard)

• Draft requirements for a CI k8s cluster in preparation for Ops to create it in the following quarter(ish)
• Nodepool deprecation: npm and php jobs, but there will be a long tail of corner case jobs
• https://i.imgur.com/2XB7Lp8.png

Program 1: Outcome 5: Objective 1: MW Nightlies server

• chad point on this, bulk of time

• Sam Reed and Chad will work on setting it up this week

Q2 goal/project check-in

All of it in table form: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Goals/201718Q2

Quarterly Goals

Program 1: Outcome 5: Milestone 1: Migrate majority of developers to JavaScript based browser test framework (webdriver.io)

Due: End of this quarter

Quarter Goal Task: Port Selenium tests from Ruby to Node.js - task T139740

• T174103 [Epic] Port Selenium tests from Ruby to Node.js for the Search Platform
  • making a list for SoS of needed migrations still
  • CirrusSearch/Discovery closed the migrate task, there are still some ruby selenium tests in the repo though, investigating :)

• T175179 Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch
  • Got smoke tests running both locally and in CI
  • JJB configuration still TODO

Ruby

• T180878 Upgrade RuboCop and Rubyzip
  • All done.

Program 3: Outcome 1: Objective 1: Define a set of code stewardship levels (from high to low expectations)

Due: End of this quarter

Quatertly Goal task: - task T174090

• Waiting to talk to Victoria/Toby

Program 3: Outcome 1: Objective 2: Identify and find stewards for high-priority/high use code segment orphans

Due: End of next quarter

Quaterly Goal task - task T174091

Program 3: Outcome 2: Objective 1: Define a “Technical Debt Project Manager” role that regularly communicates with all Foundation engineering teams regarding their technical debt

Due: End of this quarter

Program 3: Outcome 2: Objective 2: Define and implement a process to regularly address technical debt across the Foundation

Due: End of next quarter

Program 6: Outcome 2: Objective 2: Set up a continuous integration and deployment pipeline to publish new versions of an application to production via testing and staging environments that reliably reproduce production

Due: End of this quarter

Keyword: SSD

Complete build phase of release pipeline

• • Build test variant
  • Run test entrypoint w/developer feedback - services dependency
  • Build production variant w/developer feedback - services dependency
  • Tag production container
  • Push to production docker registry - ops dependency - staging namespace

Tracking: https://phabricator.wikimedia.org/T157469

current status: https://phabricator.wikimedia.org/project/view/2453/

• quarterly goal complete https://gerrit.wikimedia.org/r/#/c/395570/
• starting thinking about next quarter
  • dan working on helm config
  • tyler made a minikube host in ci (integration-slave-k8s-1002)

Quaterly non-goal "Work"

Program 1: Outcome 1: Objective 1: Scap (Tech Debt Sprint FY201718-Q2)

workboard

• scap deb build mishaps
  • scap deb was built from master (oops)
  • included the git flat dir
  • on the rebuild the quilt patches weren't applied, (oooops!)
  • rebuilt again, now works
  • documentation updates incoming

Program 1: Outcome 5: Objective 1: Maintain existing shared Continuous Integration infrastructure

Goal: A generalized POC for a docker-based CI.

https://phabricator.wikimedia.org/project/view/3008/ (shipyard workboard)

• docker-face: https://i.imgur.com/2XB7Lp8.png
• working with docker-pkg, migrating images to that build system
  • it's working more or less :)

Program 1: Outcome 6: Milestone 1: Maintain Gerrit

• Gerrit user login issue
• Upgrade planned for this week
  • scap deploy!
  • maybe... zuul is incompatible with 2.14, and hard to upgrade right now
    • blocked on https://phabricator.wikimedia.org/T158243

Program 1: Outcome 6: Milestone 2: Maintain Phabricator

• Allow CoC committee to temporarily disable Phab user accounts
  • https://phabricator.wikimedia.org/T183116

Program 1: Outcome 5: Objective 1: MW Nightlies server

• plugging along

Other work

• Follow-up work re ORES Post Mortem
  • Services review
  • Extension review

Grooming

Team Kanban Board Review and Triage

• closed and touched in the 7 days
• No update for 4 weeks
• No update for 3 weeks
• No update for 2 weeks
• No update for 1 week
• All Open
• Review To Triage column of #releng

• Assigned
• Unassigned

Once / month-ish review of backlog(s)

• releng Review To Triage column of #releng
• releng-kanban Review unassigned in kanban
• releng-kanban Review 'backlog' colum of -kanban
• releng-next - Review for things we need to put on our kanban backlog
• releng-backlog - oh my, the huge backlog of things...

Kanban stats

Burnup chart