Data Platform Engineering/Data Platform SRE/Status Update/2026-06-12
Highlights
[edit]- WDQS: purged the backlog of federation requests, also removed duplicates and updated the mediawiki.org page
- Spark on Kubernetes with Ceph/S3
Drafting A proposed security model for Data Platform services built on Kubernetes and Ceph/S3 under task T428255.
- Upgraded our Ceph cluster to the latest point release (with Balthazar) in task T428385 Multiple CVEs for the ceph package
- Added translation memory indices on test OpenSearch-on-K8S cluster, with the old CirrusSearch still working; next step is to populate indices on production K8S instance and switch it over from CirrusSearch task T425377
OpenSearch upgrade to 2.x
[edit]Support Wikidata Platform team
[edit]- T428241 Clean up of WDQS full graph endpoint (wdqs2009) Blazegraph journal file (Resolved)
- T427319 request: s3 access for wdqs::alternatives (Resolved)
SPARQL Federation requests
[edit]- T420705 Add QLever OpenStreetMap endpoint to Wikidata query service and WCQS whitelist (Resolved)
- T420702 Add https to api.finto.fi/sparql in Wikidata query service and WCQS whitelist (Resolved)
- T419205 Correct and remove outdated URLs in the Allowlist (Resolved)
Hardware
[edit]- T428582 decommission wdqs1015.eqiad.wmnet (Resolved)
- T427852 hw troubleshooting: CPU1 thermal fault for wdqs1015.eqiad.wmnet (Resolved)
Misc / Operations
[edit]- T427517 Issue 6 month certificates for OpenSearch-on-K8S (Resolved)
- T428571 Degraded RAID on an-worker1201 (Resolved)
- T418809 Package thirdparty/opensearch1 for bookworm (Resolved)
- T424381 A more recent Spark + Iceberg will make Incremental MediaWiki History much more efficient (Resolved)
- T393603 Write a blog post about the recent Airflow migration to Kubernetes (Declined)
- T428099 Airflow Dev Env Task Shell Pod Missing CLASSPATH Environment Variable (Resolved)
- T428553 Turnilo configuration is no longer updated with netbox data (Resolved)
- T428053 Deploy a kafka web UI for all internal clusters (Resolved)
- T428234 Some kafka clusters are lacking the cluster DescribeConfigs ACL (Resolved)
- T291645 Produce ECS formatted logstash logs to Event Platform, allowing them to be queried in the WMF Data Lake with SQL (Resolved)