Wikimedia Technology/Goals/2019-20 Q2

Technology Department Team Goals and Status for Q2 FY19/20 in support of the  Medium Term Plan (MTP) Priorities and Annual Plan for FY19/20



Analytics [WIP]
Team Manager: Nuria Ruiz
 * Reduce platform Complexity. Modern Event Platform
 * Build a reliable, scalable, and comprehensive platform for creating services, tools and user facing features that produce and consume event data'''
 * Resolve Kafka Connect HDFS Licensing issue and decide if we will use Kafka Connect
 * Initial (Stream) Config Service implementation in vagrant


 * Smart Tools for Better Data. Make easier to understand the history of all Wikimedia projects
 * Release Mediawiki History in JSON/CSV or mysql dump format (the best dataset to date measure content and contributors)
 * Deploy hadoop client to dump hosts so mediawiki history public dataset can get to dumps on a reasonable timeframe


 * Smart Tools for Better Data. Make easier to understand how Commons media is used across our projects.
 * Announce the deployment of the mediarequests API:
 * Add mediarequests metrics to Wikistats UI


 * Smart Tools for Better Data. Increase Data Quality, Privacy and Security
 * Deploy Enthrophy-based alarms for data issues that could indicate, bugs, traffic drops due to censorship on inconsistencies, this work continues from Q1
 * Productionize Kerberos Service
 * Create test Kerberos identities/accounts for some selected users from Analytics Team in test cluster T212258,


 * Core. Operational Excellence. Increase Resilience of Systems
 * New zookeeper cluster for tier-2


 * Core. Operational Excellence. Reduce Operational Load by Phasing Out Legacy Systems/Technologies
 * Sunset MySQL data store for eventlogging., this work continues from Q1
 * Migrate eventlogging to python3

Dependencies on:

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -



Core Platform
Team Manager: Corey Floyd

Dependencies on:

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -



Fundraising Tech
Team Manager: Erika Bjune

Dependencies on:

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -



Performance
Team Manager: Gilles Dubuc

Dependencies on:

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -



Release Engineering
Team Manager: Tyler Cipriani

Dependencies on:

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -



Quality and Test Engineering
Team Manager: JR Branaa

Dependencies on:

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -



Research
Team Manager: Leila Zia

''This section is in-progress. Please do not copy to Airtable before checking with Leila.''


 * - [P-O14-D4] A comprehensive literature review of disinformation published in arxiv and meta (completing the work started in Q1).
 * - [P-O14-D4] Build a prioritized list of actions to take (tools to build, datasets to release, etc.) for combating disinformation (though discussions with the community of editors and developers, internal consultation, and maybe with external researchers)
 * - [P-O14-D4] Build one formal collaborations in the disinformation space to start the research for building solutions starting Q3.
 * - [P-O14-D4] Prepare the Research Internship proposal.
 * - [P-O14-D4] Finalize the research brief for crosslingual topical model laying out the work that will be done in this space starting Q3.

Dependencies on:

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -



Scoring Platform
Team Manager: Aaron Halfaker
 * - Jade Entity Page UI
 * - Newcomer quality session models
 * - Expansion of Topic Model to ar, ko, and cswiki

Dependencies on:

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -



Search Platform
Team Manager: Guillaume Lederrey

Dependencies on:

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -



Security
Team Manager: John Bennett

Dependencies on:

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -



Site Reliability Engineering
Directors: Mark Bergsma and Faidon Liambotis

Dependencies on:

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -



Technical Engagement
Team Manager: Birgit Müller

Core
 * - [IaaS] All out of warranty hardware used for offsite backups of Cloud Services data in the codfw datacenter is replaced
 * - [IaaS] 60% of the remaining Debian Jessie systems in the hardware layer underlying Cloud VPS are upgraded to Debian Buster or Stretch
 * - [IaaS] All Debian Jessie instances are removed/replaced in 95% of Cloud VPS hosted projects
 * - [IaaS] Deploy a minimum viable Ceph cluster in eqiad and convert 1+ cloudvirt servers to use it for instance storage
 * - [IaaS] Measure IOPS as seen at the instance level, IOPS as seen at the Ceph cluster level, and network activity generated in delivering IOPS at the backbone network level to produce a forecast for impact of full conversion of cloudvirt servers to Ceph instance storage.
 * - [IaaS] Create a shared understanding of systems and service continuity and availability constraints in the current Cloud VPS product which can be used to design follow-on projects to reduce single points of failure and establish practices for testing and maintaining continuity and availability of Cloud VPS core services.
 * - [IaaS] OpenStack APIs and services are upgraded to the "Ocata" release
 * - [PaaS] Deploy a Kubernetes 1.15.2+ cluster in Toolforge which will be used to provide a more modern, secure, and performant PaaS baseline to Tool maintainers.
 * - [PaaS] Migrate 5+ early adopter/beta tester tools from legacy Kubernetes cluster to new Kubernetes cluster to validate integration with ingress proxy layer and sandboxing/isolation of new Kubernetes cluster deployment.
 * - [PaaS] Create timeline and operational plan for migrating all Kubernetes workloads in Toolforge to the new Kubernetes cluster and decommissioning the legacy cluster by the end of FY19/20.

Increased visibility & knowledge of technical contributions, services and consumers across the Wikimedia ecosystem (Reduce Complexity of the Platform, Movement Diversity)

Support Wikimedia's diverse technical communities (Reduce Complexity of the Platform; Movement Diversity)

Dependencies for core work is on: SRE/Data Center Operations team

 Status 
 * October 2019 status -
 * November 2019 status -
 * December 2019 status -