Docker/SIG/Meetings/20190228

2019-02-28
Lead: Greg Grossmeier Participants: 18


 * What is this group? What is a SIG? What are we trying to do? - Greg
 * Everyone here is working on something to do with containers, this meeting gives people space to talk about projects they are working on
 * Current plan is to have a meeting per month
 * Notes posted on wiki -- you are being recorded ;)


 * Description/status of the in-flight projects
 * Mark H/hexmode
 * not using a lot right now, using Jenkins docker images that WMF uses in CI, wanting to containerize the services so I can deploy for his clients
 * David Barratt
 * https://hub.docker.com/_/mediawiki / https://github.com/wikimedia/mediawiki-docker
 * Not intended for WM Production
 * Official for Docker not for MW
 * Alexandros K
 * SRE, to manage by end of next fiscal, to move to k8s for most services and MW
 * I dislike docker, doing too many things, but like as the env with k8s ontop
 * end-goal of work currently is to have alignment of production and development envs
 * Addshore
 * mediawiki-docker-dev - https://github.com/addshore/mediawiki-docker-dev
 * also have images for wikibase and wdqs - https://github.com/wmde/wikibase-docker
 * Working on a new node server side render to be deployed in production using blubber etc - https://phabricator.wikimedia.org/T212189
 * Tyler
 * Deployment Pipeline - https://wikitech.wikimedia.org/wiki/Deployment_pipeline
 * Blubber - https://wikitech.wikimedia.org/wiki/Blubber
 * Jeena Huneidi
 * Working on local development environment using Minikube
 * Minikube: https://github.com/kubernetes/minikube
 * https://gerrit.wikimedia.org/g/releng/local-charts
 * has mariadb, MW, and parsoid, currently working on Restbase
 * Haven't created any Docker images others should use, still working on that part
 * The goal here is just to make local development easy, and to get parity with deployment pipeline
 * Mateus
 * working on a maps stack for development
 * Cassandra, Redis, Tilerator
 * https://github.com/mateusbs17/gisdock


 * Discussion / Questions


 * Greg: We've got a lot of stuff - it seems like there's a lot of specific-use-case Docker work going on. Is there some possibility of us using some of the same basic foundations to build on - does anyone see potential overlap that we could start working on?
 * Jeena: Sharing base images
 * Kosta Harlan: mediawiki-docker-dev and the local-charts could probably be using the same base image for php / apache?
 * Adam: one of the biggest things people have been wanting is VE, parsoid/etc to have a real VE running
 * Adam: also EventLogging from Analytics - https://github.com/wikimedia/eventlogging/tree/master/docker (for example)
 * TODO: Greg to make the list of all in-flight things on the docker SIG wiki page
 * Tyler: random list of images in the wikimedia-docker registry, FWIW: https://people.wikimedia.org/~thcipriani/docker/
 * Jeena: list ones that have gone through some sort of approval
 * David: agreed ^, and also re the "official" images from Dockerhub are *required* to extend from the official dockerhub base images. Our images don't have that policy nor extend from those.
 * Alex: https://snyk.io/blog/top-ten-most-popular-docker-images-each-contain-at-least-30-vulnerabilities/
 * Cole: Base image lifecycle?
 * Alex: SRE is working on that a bit now, very early. Will be sharing later if you're interested
 * Tyler: wanted to share link of an occasionally (hourly-ish) updated list production docker registery: https://people.wikimedia.org/~thcipriani/docker/
 * Tyler: those are images used in production or available for use in production
 * Alex: you can build images on top of that, keep them up to date to address the security concerns linked above
 * https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/master/images/
 * Alex: you should start relying on these for things you want to see in production, feel free to use whatever else for random other non-production things you're working on
 * Tyler: docker-pkg project, from Giuseppe of SRE, manages base images for production and currently for CI
 * Antoine: docker-pkg recently received lot of updates. Give it a try!:)
 * Dan G: What criteria would something be assessed by for officilanes
 * What is meant by "official"
 * dockerhub-official vs wmf-official vs wmfproduction-official
 * plan to use same image for development as well as production
 * Eric: documentation/social project question: one of the best use cases is new developers. QUickly hit limtes of MW-V with wikibase/ES. Docker seems the way to go. The work that david and adam have been doing is super helpful, but on top of that some documentation for new people: here are some base images, here are things you can plug in. Have some good documetation for what parts can be assembled with guidance to get you what you need.
 * Brennen: there's not one clear entry point for a new developer. MW-V, docker-dev, and a bunch of people are rolling their own solution. Documentation is a solid idea. "this is the funnel"
 * TODO: Brennen and Eric Gardner volunteer to help documentation on this
 * Alex: documentation portal with at least Pipeline focused
 * Make sure those are aligned ^^^
 * Kosta: can we suppor the local-charts initiative. I'd want to put time into that if it's going to become the new dev env
 * Jeena: please do help out with the local-charts work
 * Brennen: we want feedback, can people try spinning up these charts and if they work or not etc?

Existing docs / entry points

 * https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker
 * https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker#Docker_development_environment
 * https://www.mediawiki.org/wiki/Selenium/Node.js
 * https://wikitech.wikimedia.org/wiki/Blubber
 * https://www.mediawiki.org/wiki/MediaWiki-Vagrant (not for Docker per se, but relevant to new devs)

Followups

 * TODO - Greg to add a list of all in-progress projects on the Docker SIG page
 * TODO -- some folks to look at https://gerrit.wikimedia.org/g/releng/local-charts
 * TODO - Brennen: Make an issue for followup on documentation stuff / coordinate a chat

Video call chat transcription
You 9:00 AM o/ Greg Grossmeier 9:02 AM https://etherpad.wikimedia.org/p/docker-sig David Barratt 9:06 AM https://hub.docker.com/_/mediawiki https://github.com/wikimedia/mediawiki-docker Eric Gardner 9:07 AM David, as a new developer, that image was very useful for me in getting a proper environment set up locally – was able to adapt it very easily. Thanks! David Barratt 9:09 AM Eric, Great! :) Mark A. Hershberger 9:10 AM link to node renderer addshore mentioned? blubber? Lars Wirzenius 9:10 AM https://wikitech.wikimedia.org/wiki/Blubber Antoine Musso 9:11 AM sorry long commute :( Jeena Huneidi 9:11 AM https://gerrit.wikimedia.org/g/releng/local-charts Tyler Cipriani 9:13 AM Etherpad for people just joining: https://etherpad.wikimedia.org/p/docker-sig (not sure how hangouts handles scrollback) Mateus Santos 9:13 AM http://github.com/mateusbs17/gisdock Antoine Musso 9:15 AM GO GREG GO ! :D David Barratt 9:16 AM Jeena, will the local-charts work with https://microk8s.io/ ? or just minikube? Kosta Harlan 9:16 AM mediawiki-docker-dev and the local-charts could probably be using the same base image for php / apache? Jeena Huneidi 9:17 AM David - not sure, I'll check it out! Kosta - that's possible yeah Tyler Cipriani 9:18 AM random list of images in the wikimedia-docker registry, FWIW: https://people.wikimedia.org/~thcipriani/docker/ Alexandros Kosiaris 9:19 AM https://snyk.io/blog/top-ten-most-popular-docker-images-each-contain-at-least-30-vulnerabilities/ Greg Grossmeier 9:20 AM haha Adam Shorland 9:20 AM i guess we already kind of have our own bases images to extend form? Tyler Cipriani 9:20 AM was just looking for that link :) Adam Shorland 9:21 AM
 * D

Tyler Cipriani 9:21 AM https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/master/images/ Adam Shorland 9:22 AM https://docker-registry.wikimedia.org/v2/_catalog isnt loading :( Tyler Cipriani 9:22 AM https://github.com/wikimedia/operations-docker-images-docker-pkg Alexandros Kosiaris 9:22 AM takes 30s it's very slow currently, working on upgrading to make it faster Adam Shorland 9:22 AM

Antoine Musso 9:23 AM docker-pkg recently received lot of updates. Give it a try!:) Tyler Cipriani 9:23 AM what's the definition of "official" :) Dan Garry 9:23 AM Exactly! :-D Adam Shorland 9:24 AM dockerhub-official, wmf-official, wmfproduction-official not sure either Adam Shorland 9:27 AM I really want to make the mediawiki-docker-dev thing more plugable Kosta Harlan 9:28 AM Seems like everyone finds their own way to suffer through this problem Eric Gardner 9:28 AM haha, so true You 9:28 AM perhaps the equivalent of https://www.mediawiki.org/wiki/MediaWiki-Vagrant Eric Gardner 9:28 AM yes, exactly Adam Shorland 9:28 AM i guess the page right now is https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker Tyler Cipriani 9:28 AM https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker#Docker_development_environment Adam Shorland 9:29 AM it lists vagrant and mediawiki-docker-dev or manual install [= David Barratt 9:29 AM someone created this page as well (but it needs some love) https://www.mediawiki.org/wiki/Docker Adam Shorland 9:29 AM i had no idea the mediawiki-docker-dev thing was there Kosta Harlan 9:29 AM Would be great to add more recipes there https://www.mediawiki.org/wiki/MediaWiki-Docker-Dev Mateus Santos 9:29 AM Still, the endeavors are split between several projects. Vagrant has its problems but everyone suffers together. Adam Shorland 9:30 AM dev env goals, use wmf built images in the environemt :) Jeena Huneidi 9:31 AM yeah Greg Grossmeier 9:31 AM https://www.mediawiki.org/wiki/Selenium/Node.js Tyler Cipriani 9:31 AM we ripped that off for blubber docs https://wikitech.wikimedia.org/wiki/Blubber Adam Shorland 9:33 AM I'm in another call now so cant talk but would love to try and include more services into my dev env, in a nice way where you can turn them on and off etc if you do or dont want them, if anyone has thoughts or services it might make sense for https://github.com/addshore/mediawiki-docker-dev/issues/71 is probably the place Antoine Musso 9:34 AM make sure to capture the above chat on the etherpad https://etherpad.wikimedia.org/p/docker-sig :) Eric Gardner 9:34 AM I think expanding on mediawiki-docker-dev to make it more pluggable is a good way forward even if it's just a matter of documenting how to get common setups working Tyler Cipriani 9:35 AM https://gerrit.wikimedia.org/g/releng/local-charts#readme-for-local_charts looks like there's a FIXME in MacOS if folks are interested :) Kosta Harlan 9:35 AM eek :) Adam Shorland 9:35 AM i look forward to trying local-charts over the coming week :) Kosta Harlan 9:35 AM Yeah I'll try it out Brennen Bearnes 9:35 AM cool. :)