Wikimedia Release Engineering Team/CI Futures WG/Meetings/2019-10-29-RelEng+SRE sync

ควย = 2019-10-29 =

Attendees: Giuseppe, Alex, Effie, Mark, thcipriani, dduvall, liw

Release Engineering and SRE teams create a plan to implement a Deployment Pipeline compliant CI system.

Agenda

 * Background
 * Why is it needed
 * Python3
 * SPoF (people)
 * Jenkins/security
 * CI Futures
 * More self-serve for our devs
 * Expectations from the SRE team


 * Criteria/PoCs
 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/CI_Futures_WG
 * https://www.mediawiki.org/wiki/User:LarsWirzenius/NewCI
 * Meta architecture
 * slides http://git.liw.fi/wmf-ci-arch/tree/ci-arch-summary.md
 * A: Will there be mapping from a repo -> each of these stages. Kask is an example: one repo is running two services. This repo complicates things, and will need to be supported.
 * liw: pipeline for each *end result*; i.e., if one repo is used for two services that's two pipelines
 * Effie: where are we going to start?


 * Argo/Seakeeper
 * https://argoproj.github.io/
 * Seakeeper doc: https://docs.google.com/document/d/1b6sqmfdcH4XL8wayL5OOaJX9xOyDq-osVEEkzzIWWY4/edit?usp=sharing :)
 * defines k8s CRD and responds to changes to controller
 * workflow creates outputs which can be saved as artifacts
 * could allow for self-serve
 * integration with Gerrit via Argo-events -- responds to external events


 * Questions
 * Namespaces per project -- may be too many projects
 * Gerrit has 2333 projects as of this morning


 * Machines where we will run CI


 * How stable/long-lived is argo?
 * Maintained by Intuit
 * Dan is a contributor -- is bullish on upstream
 * Tekton is slowly ripping off Argo (Tekton is in https://cd.foundation/ )


 * Security-sensitive workflows -- embargoed changes + gpg signing/docker-pkg
 * Ideally there would be a solution for that in this
 * We've been thinking about this for MediaWIki deployment re:security patches pre-deployment


 * What are the interactions with the production environment?


 * self-serve
 * Ops puppet uses rspec, tox, etc. less standard than, say, a PHP or python project
 * Document doesn't go into detail about this -- but having a small abstraction over the workflow -- analogous to .pipeline/config.yaml -- should work with well with Argo

TODO: thcipriani to followup with email, we need to have more meetings about this