Kubernetes SIG/Meetings/2023-11-28

Agenda:

• Misc:
  • Kube-state-metrics is now available and can be enabled per cluster: https://phabricator.wikimedia.org/T264625
    • First WIP dashboard at https://grafana-rw.wikimedia.org/d/WG4NjDISk/wip-cluster-status-and-capacity?orgId=1&refresh=1m
    • Drill down dashboards on a per-service basis
    • Enabled only on wikikube currently
    • Enable on “your” own clusters via admin_ng/helmfile.yaml
  • Adding new nodes to k8s clusters now requires manual “uncordon”: https://gerrit.wikimedia.org/r/c/operations/puppet/+/975258
  • PodSecurityPolicy replacement: https://phabricator.wikimedia.org/T273507
    • Replaced in k8s 1.25 with Policy Security Standard (3 fixed “classes”)
    • All workload is basically fine except for MediaWiki (because we use hostPath mounts)
    • Open Policy Agent (Gatekeeper) might be a way out, so is Validating Admission Policies.
    • MediaWiki requires hostPath (currently, for GeoIP) and PTRACE capability (for producing slow logs)
    • More research needed for this
    • Ideally we can migrate to something else decoupled from the next k8s upgrade to lower risk
  • FYI: Project to migrate Superset to DSE-K8S: T347710
    • Note to bear in mind that Superset is a critical tool for anti-DDoS response, so please don’t over-complicate it.
  • Istio images can’t be built on bookworm currently (the istio version we use does not build with go >= 1.20). Need  to stick to bullseye for now. https://phabricator.wikimedia.org/T351933