Kubernetes SIG/Meetings/2023-11-28

Agenda:


 * Misc:
 * Kube-state-metrics is now available and can be enabled per cluster: https://phabricator.wikimedia.org/T264625
 * First WIP dashboard at https://grafana-rw.wikimedia.org/d/WG4NjDISk/wip-cluster-status-and-capacity?orgId=1&refresh=1m
 * Drill down dashboards on a per-service basis
 * Enabled only on wikikube currently
 * Enable on “your” own clusters via admin_ng/helmfile.yaml
 * Adding new nodes to k8s clusters now requires manual “uncordon”: https://gerrit.wikimedia.org/r/c/operations/puppet/+/975258
 * PodSecurityPolicy replacement: https://phabricator.wikimedia.org/T273507
 * Replaced in k8s 1.25 with Policy Security Standard (3 fixed “classes”)
 * All workload is basically fine except for MediaWiki (because we use hostPath mounts)
 * Open Policy Agent (Gatekeeper) might be a way out, so is Validating Admission Policies.
 * MediaWiki requires hostPath (currently, for GeoIP) and PTRACE capability (for producing slow logs)
 * More research needed for this
 * Ideally we can migrate to something else decoupled from the next k8s upgrade to lower risk
 * FYI: Project to migrate Superset to DSE-K8S: T347710
 * Note to bear in mind that Superset is a critical tool for anti-DDoS response, so please don’t over-complicate it.
 * Istio images can’t be built on bookworm currently (the istio version we use does not build with go >= 1.20). Need  to stick to bullseye for now. https://phabricator.wikimedia.org/T351933