Wikimedia Release Engineering Team/Deployment pipeline/2019-05-09

Last Time

 * 2019-04-11
 * Archive

General
—

TODOs from last time

 * TODO what are our annual plans WRT to this project


 * TODO various attack vectors document to start


 * TODO: support documention like the one tyler did for the portal and pipeline/helmfile and deployment
 * https://wikitech.wikimedia.org/wiki/Deployment_pipeline now exists, https://wikitech.wikimedia.org/wiki/Continuous_Delivery has been deleted.
 * reached out to technical writer, in very preliminary stages for next quarter
 * think about other documentation needs beyone Deployment_pipeline main page
 * Anything we can do here without stepping on toes?
 * preliminary research happening
 * work with srodlund about what the portal is
 * starting on the real work of the documentation next week


 * TODO docs for service docker container in beta


 * ✅ TODO: announce email
 * https://lists.wikimedia.org/pipermail/wikitech-l/2019-April/091976.html
 * Feedback?
 * Nothing major, mostly queries about MediaWiki containerisation.
 * Some talk about Migrate Beta cluster services to use Kubernetes

RelEng

 * pipelinelib review meeting after this -- hopefully make progress towards .pipeline/config.yaml
 * 502918 - pipeline: Builder and stage implementation
 * 502919 - pipeline: Provide a rickety but useful system test
 * 502917 - pipeline: Directed graph execution model
 * Docs are published for pipelinelib
 * https://doc.wikimedia.org/pipelinelib/
 * deployment to staging -- we deploy to ci namespace on neon, but we also run helm delete
 * to deploy to staging in the general case: deploy to staging with a namespace that is the same name as the service
 * problems and edgecases with this are repos with multiple services that need to be deployed; e.g., eventgate-ci
 * pipelinelib could potentially allow for multiple services to be deployed from a single repository; i.e., .pipeline/config.yaml
 * for helm test, consider using:
 * https://github.com/kubernetes-sigs/kind


 * Local development images using blubber:
 * https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/508439
 * https://gerrit.wikimedia.org/r/c/mediawiki/core/+/508392

Serviceops

 * kask got swagger/openapi spec. Merge pending
 * how do we do this for staging since kask needs cassandra?
 * setup vms? we have hardware possibly...
 * citoid has similar problems, but if we could pass a proxy to the helm test
 * Dan: we could provide configuration to pass to helm install or helm test in the pipeline
 * something to think about: things we have no plan to move to k8s (e.g., mysql) how do we deal with those in staging -- resource problems in general
 * maybe possible to work around the helm test part with some hack
 * e.g., restbase works around this by heavily mocking responses
 * this may be harder with MediaWiki
 * quibble currently works around this
 * MediaWiki has this problem in the testrunner a bit i.e., unit tests fail if it tries to write to the database -- other things write to sqlite
 * stress testing for projects
 * staging, in theory, in the future should be able to be used for some portion of stress testing
 * two problems: (1) providing servers for data (2) providing the data -- the 2nd one will be difficult
 * "staging"
 * is this a persistent k8s pre-production environment
 * is this a environment where we could mock important services
 * these two things serve two different testing strategies


 * some work on making kask more easily working on minikube, courtesy of incubator/cassandra chart
 * Tech talk scheduled from July 10
 * new registryha live, no complains so far
 * https://phabricator.wikimedia.org/T209271

Services
—

= As Always =
 * Release Pipeline Workboard
 * Meeting notes