Docker/SIG/Meetings/20190228

2019-02-28

Lead: Greg Grossmeier Participants: 18

What is this group? What is a SIG? What are we trying to do? - Greg
- Everyone here is working on something to do with containers, this meeting gives people space to talk about projects they are working on
- Current plan is to have a meeting per month
- Notes posted on wiki -- you are being recorded ;)

Description/status of the in-flight projects
- Mark H/hexmode
  - not using a lot right now, using Jenkins docker images that WMF uses in CI, wanting to containerize the services so I can deploy for his clients
- David Barratt
  - https://hub.docker.com/_/mediawiki / https://github.com/wikimedia/mediawiki-docker
  - Not intended for WM Production
  - Official for Docker not for MW
- Alexandros K
  - SRE, to manage by end of next fiscal, to move to k8s for most services and MW
  - I dislike docker, doing too many things, but like as the env with k8s ontop
  - end-goal of work currently is to have alignment of production and development envs
- Addshore
  - mediawiki-docker-dev - https://github.com/addshore/mediawiki-docker-dev
  - also have images for wikibase and wdqs - https://github.com/wmde/wikibase-docker
  - Working on a new node server side render to be deployed in production using blubber etc - https://phabricator.wikimedia.org/T212189
- Tyler
  - Deployment Pipeline - https://wikitech.wikimedia.org/wiki/Deployment_pipeline
  - Blubber - https://wikitech.wikimedia.org/wiki/Blubber
- Jeena Huneidi
  - Working on local development environment using Minikube
    - Minikube: https://github.com/kubernetes/minikube
  - https://gerrit.wikimedia.org/g/releng/local-charts
  - has mariadb, MW, and parsoid, currently working on Restbase
  - Haven't created any Docker images others should use, still working on that part
  - The goal here is just to make local development easy, and to get parity with deployment pipeline
- Mateus
  - working on a maps stack for development
  - Cassandra, Redis, Tilerator
  - https://github.com/mateusbs17/gisdock

Discussion / Questions

- Greg: We've got a lot of stuff - it seems like there's a lot of specific-use-case Docker work going on. Is there some possibility of us using some of the same basic foundations to build on - does anyone see potential overlap that we could start working on?
  - Jeena: Sharing base images
  - Kosta Harlan: mediawiki-docker-dev and the local-charts could probably be using the same base image for php / apache?
- Adam: one of the biggest things people have been wanting is VE, parsoid/etc to have a real VE running
- Adam: also EventLogging from Analytics - https://github.com/wikimedia/eventlogging/tree/master/docker (for example)
- TODO: Greg to make the list of all in-flight things on the docker SIG wiki page
- Tyler: random list of images in the wikimedia-docker registry, FWIW: https://people.wikimedia.org/~thcipriani/docker/
- Jeena: list ones that have gone through some sort of approval
- David: agreed ^, and also re the "official" images from Dockerhub are *required* to extend from the official dockerhub base images. Our images don't have that policy nor extend from those.
- Alex: https://snyk.io/blog/top-ten-most-popular-docker-images-each-contain-at-least-30-vulnerabilities/
- Cole: Base image lifecycle?
- Alex: SRE is working on that a bit now, very early. Will be sharing later if you're interested
- Tyler: wanted to share link of an occasionally (hourly-ish) updated list production docker registery: https://people.wikimedia.org/~thcipriani/docker/
- Tyler: those are images used in production or available for use in production
- Alex: you can build images on top of that, keep them up to date to address the security concerns linked above
  - https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/master/images/
- Alex: you should start relying on these for things you want to see in production, feel free to use whatever else for random other non-production things you're working on
- Tyler: docker-pkg project, from Giuseppe of SRE, manages base images for production and currently for CI
- Antoine: docker-pkg recently received lot of updates. Give it a try!:)
- Dan G: What criteria would something be assessed by for officilanes
  - What is meant by "official"
  - dockerhub-official vs wmf-official vs wmfproduction-official
  - plan to use same image for development as well as production
- Eric: documentation/social project question: one of the best use cases is new developers. QUickly hit limtes of MW-V with wikibase/ES. Docker seems the way to go. The work that david and adam have been doing is super helpful, but on top of that some documentation for new people: here are some base images, here are things you can plug in. Have some good documetation for what parts can be assembled with guidance to get you what you need.
- Brennen: there's not one clear entry point for a new developer. MW-V, docker-dev, and a bunch of people are rolling their own solution. Documentation is a solid idea. "this is the funnel"
- TODO: Brennen and Eric Gardner volunteer to help documentation on this
- Alex: documentation portal with at least Pipeline focused
- Make sure those are aligned ^^^
- Kosta: can we suppor the local-charts initiative. I'd want to put time into that if it's going to become the new dev env
- Jeena: please do help out with the local-charts work
- Brennen: we want feedback, can people try spinning up these charts and if they work or not etc?

Existing docs / entry points

- https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker
- https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker#Docker_development_environment
- https://www.mediawiki.org/wiki/Selenium/Node.js
- https://wikitech.wikimedia.org/wiki/Blubber
- https://www.mediawiki.org/wiki/MediaWiki-Vagrant (not for Docker per se, but relevant to new devs)

Followups

TODO - Greg to add a list of all in-progress projects on the Docker SIG page
TODO -- some folks to look at https://gerrit.wikimedia.org/g/releng/local-charts
TODO - Brennen: Make an issue for followup on documentation stuff / coordinate a chat

Video call chat transcription

You
9:00 AM
o/
Greg Grossmeier
9:02 AM
https://etherpad.wikimedia.org/p/docker-sig
David Barratt
9:06 AM
https://hub.docker.com/_/mediawiki
https://github.com/wikimedia/mediawiki-docker
Eric Gardner
9:07 AM
David, as a new developer, that image was very useful for me in getting a proper environment set up locally – was able to adapt it very easily. Thanks!
David Barratt
9:09 AM
Eric, Great! :)
Mark A. Hershberger
9:10 AM
link to node renderer addshore mentioned?
blubber?
Lars Wirzenius
9:10 AM
https://wikitech.wikimedia.org/wiki/Blubber
Antoine Musso
9:11 AM
sorry long commute :(
Jeena Huneidi
9:11 AM
https://gerrit.wikimedia.org/g/releng/local-charts
Tyler Cipriani
9:13 AM
Etherpad for people just joining: https://etherpad.wikimedia.org/p/docker-sig (not sure how hangouts handles scrollback)
Mateus Santos
9:13 AM
http://github.com/mateusbs17/gisdock
Antoine Musso
9:15 AM
GO GREG GO ! :D
David Barratt
9:16 AM
Jeena, will the local-charts work with https://microk8s.io/ ? or just minikube?
Kosta Harlan
9:16 AM
mediawiki-docker-dev and the local-charts could probably be using the same base image for php / apache?
Jeena Huneidi
9:17 AM
David - not sure, I'll check it out!
Kosta - that's possible yeah
Tyler Cipriani
9:18 AM
random list of images in the wikimedia-docker registry, FWIW: https://people.wikimedia.org/~thcipriani/docker/
Alexandros Kosiaris
9:19 AM
https://snyk.io/blog/top-ten-most-popular-docker-images-each-contain-at-least-30-vulnerabilities/
Greg Grossmeier
9:20 AM
haha
Adam Shorland
9:20 AM
i guess we already kind of have our own bases images to extend form?
Tyler Cipriani
9:20 AM
was just looking for that link :)
Adam Shorland
9:21 AM
:D
Tyler Cipriani
9:21 AM
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/master/images/
Adam Shorland
9:22 AM
https://docker-registry.wikimedia.org/v2/_catalog isnt loading :(
Tyler Cipriani
9:22 AM
https://github.com/wikimedia/operations-docker-images-docker-pkg
Alexandros Kosiaris
9:22 AM
takes 30s
it's very slow currently, working on upgrading to make it faster
Adam Shorland
9:22 AM
:(
Antoine Musso
9:23 AM
 docker-pkg recently received lot of updates. Give it a try!:)
Tyler Cipriani
9:23 AM
what's the definition of "official" :)
Dan Garry
9:23 AM
Exactly! :-D
Adam Shorland
9:24 AM
dockerhub-official, wmf-official, wmfproduction-official
not sure either
Adam Shorland
9:27 AM
I really want to make the mediawiki-docker-dev thing more plugable
Kosta Harlan
9:28 AM
Seems like everyone finds their own way to suffer through this problem
Eric Gardner
9:28 AM
haha, so true
You
9:28 AM
perhaps the equivalent of https://www.mediawiki.org/wiki/MediaWiki-Vagrant
Eric Gardner
9:28 AM
yes, exactly
Adam Shorland
9:28 AM
i guess the page right now is https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker
Tyler Cipriani
9:28 AM
https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker#Docker_development_environment
Adam Shorland
9:29 AM
it lists vagrant and mediawiki-docker-dev or manual install
[=
David Barratt
9:29 AM
someone created this page as well (but it needs some love) https://www.mediawiki.org/wiki/Docker
Adam Shorland
9:29 AM
i had no idea the mediawiki-docker-dev thing was there
Kosta Harlan
9:29 AM
Would be great to add more recipes there https://www.mediawiki.org/wiki/MediaWiki-Docker-Dev
Mateus Santos
9:29 AM
Still, the endeavors are split between several projects. Vagrant has its problems but everyone suffers together.
Adam Shorland
9:30 AM
dev env goals, use wmf built images in the environemt :)
Jeena Huneidi
9:31 AM
yeah
Greg Grossmeier
9:31 AM
https://www.mediawiki.org/wiki/Selenium/Node.js
Tyler Cipriani
9:31 AM
we ripped that off for blubber docs
https://wikitech.wikimedia.org/wiki/Blubber
Adam Shorland
9:33 AM
I'm in another call now so cant talk
but would love to try and include more services into my dev env, in a nice way where you can turn them on and off etc if you do or dont want them, if anyone has thoughts or services it might make sense for https://github.com/addshore/mediawiki-docker-dev/issues/71 is probably the place
Antoine Musso
9:34 AM
make sure to capture the above chat on the etherpad https://etherpad.wikimedia.org/p/docker-sig :)
Eric Gardner
9:34 AM
I think expanding on mediawiki-docker-dev to make it more pluggable is a good way forward
even if it's just a matter of documenting how to get common setups working
Tyler Cipriani
9:35 AM
https://gerrit.wikimedia.org/g/releng/local-charts#readme-for-local_charts looks like there's a FIXME in MacOS if folks are interested :)
Kosta Harlan
9:35 AM
eek :)
Adam Shorland
9:35 AM
i look forward to trying local-charts over the coming week :)
Kosta Harlan
9:35 AM
Yeah I'll try it out
Brennen Bearnes
9:35 AM
cool. :)