Data Platform Engineering/Data Platform SRE/Status Update/2026-06-19
Highlights
[edit]- Kafka: Reacted after an incident where a single topic / partition filled 3 brokers. All large topics now have more partitions, to ensure each is under 30GB. task T429127
- Fundraising Tech: Enabled airflow-fr-tech to egress to the fr-tech s3 endpoint task T428294 & task T429048
- Growthbook / Test Kitchen: Provisioned API keys in secrets allowing integration between TK and Growtbook, as well as TK and phabricator task T428985 & task T428986
- Hadoop to K8S/Ceph Migration: More work drafting: A proposed security model for Data Platform services built on Kubernetes and Ceph/S3
- Wikidata Platform Team support: Brought the 7 new WDQS nodes into service as k8s workers
- OpenSearch upgrade: ToolHub is switched over to OpenSearch on k8s task T426073, Write lightweight OCI-image-based Puppet plans for beta cluster task T425585
- Kafka: Cleaned up unused dev/test topics on kafka-jumbo after verifying they were empty task T427951
Wikidata Platform
[edit]- T403627 wikidata blazegraph journal file (Resolved)
- T423312 Q4:rack/setup/install dse-k8s-wdqs200[1-4 (formerly wdqs20[28-31])] (Resolved)
- T423314 Q4:rack/setup/install dse-k8s-wdqs100[1-3 (formerly wdqs103[6-8])] (Resolved)
- T428455 wdqs-backend-init: Support for S3 bucket sub-path (Resolved)
OpenSearch upgrade
[edit]- T426073 Migrate toolhub indices from production OpenSearch to OpenSearch on k8s (Resolved)
- T421763 Migrate beta cluster to OpenSearch 2.x (Duplicate)
GrowthBook / Test Kitchen
[edit]- T428986 [GrowthBook Integration Add write API key for Phabricator for Test Kitchen] (Open)
FR Tech
[edit]- T429048 Add FR Tech s3 secrets to airflow vars (Resolved)
- T428294 Allow egress from airflow workers to fr-tech minio (Resolved)
Misc / Operations
[edit]- T419992 Alert if calico BGP sessions are not established on any kubernetes worker (Duplicate)
- T426589 Alert in need of triage: ResourceQuotaMemoryLimitsWarning (Resolved)
- T428385 Multiple CVEs for the ceph package (Open)
- T429088 staging.webrequest.page_view.dev0 taking up most space on kafka-jumbo (Resolved)
- T428644 analytics-test gobblin tasks failure (Resolved)