Wikimedia Release Engineering Team/Deployment pipeline/20170725-planning

2017-07-25

Existing meetings

• Weekly tactical - (all meeting notes: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Deployment_pipeline )
• Monthly Mark/Kevin (hopefully can be replaced by this one)

• Phabricator task hierarchy: https://phabricator.wikimedia.org/T170453This meeting will be monthly

Purpose of this meeting: Longer term. What is needed during the upcoming month. 

Team-by-team status

Ops

• WIthin last couple quarters
  • Fixed Puppetization
  • Created production clusters
  • Created staging cluster
  • Worked a bit on base images and how to manage docker security updates in production (ideas)
• This quarter
  • Kubernetes: Try to upgrade to 1.7; trying to keep 2 clusters in sync
  • Alex working on network isolation and pods
  • Try to define standard structure (container plus helper containers)
• Current/upcoming dependencies
  • Not at the moment. Will work with cloud on Kubernetes upgrade. 
  • Will want feedback after the plan is written

Services

• Created basic minikube test cluster: https://github.com/wikimedia/mediawiki-containers/blob/master/README.k8s.md
  • Started work on `mwctl` utility to manage edit -> test workflow: https://github.com/wikimedia/mwctl
  • Design notes at https://docs.google.com/document/d/1Z1nPkWea-YtWoYAvHVK8SNoD7gM8yRGf_VbiRkWalsw/edit#
• RFC on container structure standards: https://phabricator.wikimedia.org/T169998
• Marko working on documenting use cases
• Next step: Make it a product
  • mwctl tool: allow you to test and develop code in minikube environment
  • RFC - https://phabricator.wikimedia.org/T169998 (see also above)
    • Status of RFC: Parts don't matter, but waiting on us (in this call) to agree on them
    • mwctl relies on conventions. So ideally resolve the RFC "soon" (the sooner the better)
• Blocked/dependencies
  • Document use cases (Marko will write and circulate)

Release Engineering

• Worked on abstraction to generate images for various stages of pipeline (dev/staging/prod)
  • https://phabricator.wikimedia.org/source/blubber/
• review of acl for jenkins for a where to build these containers for pushing into staging pipeline
  • https://phabricator.wikimedia.org/T169557 (could use more review)
• Dan starting review of deployment/k8s packaging stuff with helm -- to sync up with services work
  • relates to mwctl work
• Dependencies
  • acl is the big thing (we have discussed at weekly meetings)
  • Need a bit more sync-up with ops about orchestration tools (e.g. helm)...will be blocked early next quarter

Next month, this meeting should start to think about Q2 goals for the program

Program status

https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Final/Programs/Technology#Program_6._Streamlined_service_delivery Outcomes, Objectives, and Milestones Outcome 1: We have seamless productization and operation of (micro)services.

• Objective 1: Set up production-ready Kubernetes cluster(s) with adequate capacity
  • Done (can expand capcity as needed with budget)
• Objective 2: Create a standardized application environment for running applications in Kubernetes
  • Working on it this quarter (and next)

Outcome 2: Developers are able to develop and test their applications through a unified pipeline towards production deployment.

• Objective 1: Create guidelines and abstractions for building and testing applications in containers
  • Work ongoing by Ops (but not tied to a quarterly goal)
• Objective 2: Set up a continuous integration and deployment pipeline to publish new versions of an application to production via testing and staging environments that reliably reproduce production
  • Quarterly goal by RelEng
  • Includes Blubber
  • Define functional tests for Mathoid running on the staging Kubernetes cluster for use in future gating decisions - task T170482
  • Define method for monitoring and reacting to the above functional tests - task T170483
• Objective 3: Provide a lightweight integrated development environment that lets developers test their code against a local miniature copy of the production stack
  • Work ongoing by Services; prototype ready. (but not tied to a quarterly goal)

https://www.mediawiki.org/wiki/User:KSmith_(WMF)/Technology_planned_program_work_Q1#Technology_Program_6._Streamlined_service_delivery

Discussion about work tracking in phab

• Work should be tracked to programs/outcomes/objectives/q-goals
• Seems most useful to tie to objectives
• ACTION: Everyone (teams) draft goals for Q2 by next meeting
• ACTION: Mark will try to figure out a single way for all work within this program to track work