Wikimedia Release Engineering Team/MediaWiki on Kubernetes/Meeting notes/2020-12-02

= 2020-12-02 =

Always

 * Core_Platform_Team/Initiatives/MediaWiki_on_Kubernetes
 * Wikimedia_Release_Engineering_Team/MediaWiki_on_Kubernetes
 * Workboard
 * IRC:

TODOs from last time

 * SRE: Base images for fcgi and apache merged but not yet published => done
 * PET: Is there an ETA on shellbox? => 3 months!

RelEng

 * Ahmon: Started working Localisation, first experiments, startup time would be about ~30 seconds on a fresh (no existing CDBs) node.
 * Jeena: working w/ Ahmon, + Shellbox
 * Dan: experimenting on building MW (T268368). We want people to have control over building via the repo.
 * Write up of BuildKit based method here https://phabricator.wikimedia.org/T268368
 * Ran into https://github.com/moby/buildkit/issues/1822
 * Jeena: May we need a seperate registry for images with security patches?
 * Alex: we have had some bad ideas, but nothing concrete. We could store that in a layer, but we can't have private layers.
 * Dan: We have talked about building MW config as an image, which should be stored somewhere before deployment, can we have a 2nd registry?
 * Alex: how can we distinct the?
 * Effie: what if we have 2 registries where one syncs from the other (and we keepthe security stuff on the private one)?
 * Ahmon: Two registries. Public and private.  When building a security-patched MW image, pull base images from the public registry, apply patches and private config, and push to the private registry.
 * Alex: that could work, not optimal, but it could
 * Dan: We can have an image of MW config which will go to the public one, and then have an extra step where we add all the sensitive things
 * alex: How would be automate having 2 registries? how the UX would be like?
 * Effie: if we can come up with a list of requirements, we can see if we can develop the missing part that will bridge those 2 registries and improve UX
 * Jeena: Is it okay to put image tags for a private docker registry in a publicly available place, like the deployment-charts repo?
 * Alex: depends on what we want to let the users know, but probably okay

Serviceops

 * Alex: Base images are up, apache + php-fpm
 * Alex: We have an issue with logs. Apache logs to files, so now we have the issue of how to manage the access logs. We are talking to observability and see if we can push those to logstash. The rate is pretty high. We might have to fundamentally change how we do logging

Platform Engineering

 * Cindy: Shellbox is in security review (T268092), Tim estimates it will go live possibly in 3 months.
 * Cindy: Dell is trying to implement a mediawiki installation that is similar to english wikimedia, they would like to get in touch with us for building docker images.
 * Alex: -- if they are planning to open source what they will do, it will be fine to talk to them and exchange ideas. We could do an knowledge exchange --

TODOs for next time

 * RelEng: Outline model for security patch application and deployment
 * There was discussion about having a private registry, but when is that pushed to?