Data Platform Engineering/Data Platform SRE/Status Update/2026-07-10

Highlights

• Dumps: We've gone through a large stabilization effort to ensure the dump DAGs can soon be managed by ops week
• Kubernetes upgrade: We upgraded our cloudnative pg operator to support a future Kubernetes upgrade to 1.34
• Datahub: Upgraded https://datahub.wikimedia.org/ to the latest version. Still stabilising after a complete rebuild of the OpenSearch indices took longer than expected.
• DDoS mitigation infrastructure: Wrote a proposal for a simpler and lighter webrequest_sampled_live pipeline and there was much high-quality discussion arising from this.
• Hadoop / OS upgrade: Ongoing work to ensure that our Hadoop stack works on bookworm. Template:PHabT
• Ceph Security Model: More discussion with stakeholders on the security model for the Data Platform S3/Ceph services and our strategic plans for a post-Hadoop infrastructure. task T428255
• OpenSearch upgrade: Migration is complete! We have a number of learnings that need to be documented.
• ML / IPIP: Upgraded ML cluster to use IPIP routing

Wikidata Platform support

• T424338 Provide a helm chart and dse-k8s helmfiles for wdqs-proxy. (Resolved)
• T426764 Airflow secret for S3 credentials for wikidata-platform (Resolved)
• T424340 QLever db node setup for WDQS v2 (Resolved)
• T428925 DNS entries for the WDQS deployments (Resolved)
• T420727 New k8s service request: wdqs-proxy (Resolved)
• T429313 Ingress configuration for the wdqs(-next) DSE namespaces (Resolved)
• T429331 Import and adapt the upstream TopoLVM Helm chart (Resolved)
• T427348 S3 credentials for the WDQS K8S deployment (Resolved)

Alerting

• T431306 SLOBudgetBurn (Duplicate)
• T429975 AlertLintProblem (Resolved)
• T429915 AlertLintProblem (Resolved)
• T429422 AlertLintProblem (Resolved)
• T429363 AlertLintProblem (Resolved)
• T429010 AlertLintProblem (Resolved)
• T428616 AlertLintProblem (Resolved)
• T427911 AlertLintProblem (Resolved)
• T414413 Alert in need of triage: KubernetesAPIErrorRate (Resolved)

Experimentation Platform

• T430024 Allow egress from test kitchen to the phabricator API (Resolved)
• T431015 GrowthBook staging sluggish and throws authentication errors (Resolved)

SPARQL Federation

• T407109 Add https://standaarden.overheid.nl/tooi/sparql to WDQS allowlist (Resolved)
• T430523 Add https://lgbtdb.wikibase.cloud/query/sparql to WDQS allowlist (Resolved)
• T430264 Add https://linked.rism.io/api to WDQS allowlist (Resolved)

Misc / Operations

• T353481 Publish Elastic-related packages for Bookworm (Declined)
• T353392 Ensure Elastic stack works on bookworm (Declined)
• T430910 codfw: rack B4 maintenance (Resolved)
• T431112 The webrequest_sampled_live data pipeline and its query tools have become mission-critical and require re-engineering for resilience (Declined)
• T421721 Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets (Resolved)
• T431057 Ensure prometheus-blazegraph-exporter works on Debian Bookworm or later (Resolved)
• T420438 Migrate ML k8s apiserver and services to IPIP (Resolved)
• T424493 Upgrade the cloudnative-pg operator to a version supporting Kubernetes 1.31 (Resolved)
• T430346 Investigate unusually high QPS values for an-mariadb1001 (Resolved)