Wikimedia Release Engineering Team/SSD Sync Up/2019-08-06

2019-08-06

Last Time: 2019-07-30 Goals: https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2019-20_Q1#Release_Engineering

Deployment Pipeline

Workboard

GOALS

• Migrate restrouter
  • https://phabricator.wikimedia.org/T223953
• (Stretch): MobileContentService
• (Stretch): Preparatory MediaWiki config clean-up & static loading work 
  • Moving configuration to static configuration (e.g., json files rather than a bunch of PHP)

TODOs from Last Time

• N Not done TODO Need to fix pipelinelib's base image -- make a task
• N Not done TODO Add a changelog/NEWS file to blubber

• stalled brennen -- pipeline config validation
  • now unlicked

New CI

GOALS

• POCs of GitLab, Argo, and Zuul3 systems; evaluate options
  •  Done GitLab probably done
    • ```TODO``` Lars: write up GitLab PoC evaluation
    •  Done Lars to set up meeting to discuss evaluation
      • • Not done

• • Dan working on argo
    • parts of PoC working
    • worked with upstream to implement more features, this time for git artifacts (both merged but not released)
    • Moving with PoC running master
    • Argo "workflow" k8s resource that defines task to run -- source that from the target repo as a PoC
    • future work: transform .pipeline/config.yaml to workflow definition
    • Argo Events to report back to gerrit...maybe
    • Patched workflow controller instead of all components
    • Lars: Not necessary to run upstream releases for PoC

• • Brennen working on zuulv3
    • In progress
    • (Still) Highly dependent on train
    • James: is there an update about how easy it would be to migrate existing zuulv2 work to zuulv3?
    • Brennen: my sense is a lot of work but doable (based on talking to Antoine) -- I have a feeling that it is a good deal less work than migrating to another system
    • James: is the zuul migration like one we would end up building from scratch? Aka are we stacking the deck in favour of zuulv3 against argo and GitLab by doing this?
    • Tyler: We're likely to build with nodepool using a static driver to map to the current jenkins agents; if we were doing it "properly" it'd be based on k8s/whatever.
    • Dan: we should consider how difficult it is to migrate specific jobs in the evaluation
    • Brennen: Worth trying k8s with zuul / nodepool?
    • Lars: Pipelinelib written in groovy cannot be used outside of jenkins, so we should consider how difficult this will be to migrate
    • James: We have 423 JJB jobs + 8500+ of lines of layout for ~1400 specific repos -- although they're maybe not "good" -- migration is a chance to fix things
    • Dan: if pipelinelib were the new way to do things, we should evaluate how easy that is to implement in the different systems

• Document an implementable architecture for what we want in new CI
  • Lars: Will have new version hopefully for Thursday (didn't, still working on this)

Local Development

GOALS

• Migrate local-charts to deployment-charts
• Instantiate testing and linting of helm charts
• Preliminary work on a CLI for setup/management
  • Progressing slowly: https://gerrit.wikimedia.org/r/c/releng/local-charts/+/525563

TODOs from Last Time

• mediawiki/core blubber
  •  Done TODO add branch filter for this job to only trigger on master
  • Ready for review? https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/525972/
  • Blubber doesn't allow modification of files
    • runs: {insecurely: true}

Other Work

• Porting from local-charts to deployment-charts
  • In progress MediaWiki
    •  Done Move mediawiki chart from local-charts to deployment-charts repo
      • https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/522584
        • Wasn't this blocked on? (no, we will update the chart after that work is done)
          • In progress Move dev-images PHP image from php -S to Apache + php-fpm
            • https://gerrit.wikimedia.org/r/#/c/releng/dev-images/+/525842/
            • marked WIP
    • Update local-charts repository to use mediawiki chart from deployment-charts repo Template:Has patchset
      • Mukunda left feedback

• • In progress Restbase
    • Move restbase chart from local-charts to deployment-charts repository
      • Add restbase chart (port from local-charts)
    • Update local-charts repository to use restbase chart from deployment-charts repo

• • In progress Parsoid
    •  Done Move parsoid chart from local-charts to deployment-charts repository
    • Template:Patchset https://gerrit.wikimedia.org/r/c/releng/local-charts/+/527860 Update local-charts repository to use parsoid chart from deployment-charts repo

• Local Dev CLI
  • Mukunda has updated patchset
  • Refactoring -- expect patchset update later today