Wikimedia Release Engineering Team/CI Evaluation Cabal/2019-09-17

From MediaWiki.org
Jump to navigation Jump to search

Argo[edit]

tl;dr: I think argo could work for us, but there are a few considerations/caveats

  • Maintenance - not super clear how many people are involved long term
    • Two contribs from Intuit
    • MinIO -- Argo works with a bunch of different artifact stores
      • Some kind of artifact store is probably necessary, stuff is ephemeral in general
      • Build logs are separate from artifact store - k8s pod logs
    • Argo Events/Argo Core
  • Knative - relies 100% on Kubernetes
  • Deeply embedded into whatever k8s it's installed inside
    • Uses CRDs -- like a resource (e.g., pods, resources) -- but your own design -- use the CRUD api and install things onto the controller to act on resources
    • CRDs use k8s workload scheduing
  • CRDs
    • Workflows
      • Listens to changes, kicks off pods to execute workflows
    • Events
      • consumes external events and kicks off processes within k8s
      • e.g., setup a gateway that exposes a webhook endpoint


  • Q: Could this be installed in its own namespace in an existing cluster?
    • Yes; it responds to things cluster-wide...
    • I think it'd work in our existing cluster.
  • Pipelinelib
    • an unknown, but a possibility
    • Argo Events to kickoff a TBD (possibly a new CRD to be definied)
  • Gateway can override anything in the spec file
  • Argo expects a bunch of inputs and outputs
    • each task operates on inputs that are outputs of previous tasks
    • Parameters == name/value pairs, operate without artifact store
    • Artifacts == artifacts in the artifact store
  • Bus Factor
    • 2 core contributors
    • Both work at Intuit
    • it's low
  • Release Engineering invests in Go is a precondition of this
    • But RelEng will probably invest in Go anyway
  • Rollout
    • Gerrit reporting
    • Merger/Dependent pipeline logic
    • Exposing workflows vs using pipelinelib
    • Porting JJB -> workflows *could be done*
  • Filters in Argo using Kafka plugin in Gerrit