Wikimedia Technology/Annual Plans/FY2019/TEC2: Modern Event Platform/Goals

=Program Goals and Status for FY18/19=

TEC2: Modern Event Platform
 * Goal Owner: Nuria Ruiz
 * Program Goals for FY18/19: A modern event data platform will make it easier for engineers to build infrastructure for Knowledge as a Service. It will enable measuring the effectiveness of engineering projects, and also provide a base for smart reactive services, such as dependency tracking.
 * Annual Plan: TEC2: Modern Event Platform
 * Primary Goal is Knowledge as a Service: Evolve our systems and structures
 * Tech Goal: Sustaining



 = Q1 Goals =

Outcome 1 / Output 1.1 - 1.4
Wikimedia engineers have a reliable, scalable, and comprehensive platform for building services that produce and consume event data for analytics and production.
 * Events can easily and reliably be produced by internal and external clients and consumed by other internal services.

Dependencies on: Analytics, Services, SRE

Goal

 * TechCom RFCs underway and technical decisions made. (more)

Status
July 2018
 * Discussed JSONSchema versus AVRo and decision was taken to use JSONSchemas and is ✅

August 2018
 * Discussed schema registry and metadata service
 * Discussed scalable event intake service

September 18, 2018
 * RFC for schema registry closing soon. We will leave metadata service out of MVP. We will work on scalable event intake as part of next quarter goals.



=Q2 Goals =

Outcome 1 / Output 1.1
Wikimedia engineers have a reliable, scalable, and comprehensive platform for building services that produce and consume event data for analytics and production.
 * Events can easily and reliably be produced by internal and external clients and consumed by other internal services.

Dependencies on: Core Platform Team, SRE

Goal

 * Development of intake service for events whose transport is JSONSchema/http

Status
October 19, 2018
 * Planing on what language/platform we are going to be building the intake service ✅

November 14, 2018
 * Intake service prototype is being built in node ✅

December 12, 2018
 * Code can be found here: https://github.com/wikimedia/eventgate ✅



=Q3 Goals =

Outcome 1 / Output 1.1
Wikimedia engineers have a reliable, scalable, and comprehensive platform for building services that produce and consume event data for analytics and production.


 * Events can easily and reliably be produced by internal and external clients and consumed by other internal services.

Dependencies on: Core Platform Team, SRE, Release Engineering

Goal(s)

 * Deployment of Stream Intake Service (AKA EventGate) for production events using TEC3: Deployment Pipeline.
 * Mediawiki Monolog+Kafka usage migrated to EventGate.
 * Migration of some mediawiki 'Eventbus' events to EventGate (stretch goal).
 * Decomission old 'analytics' Kafka cluster. (stretch goal).

Status
January 2019
 * Deployment via Docker & Kubernetes in beta and then production

February 2019
 * Produce Monolog based events from Mediawiki to EventGate and create Hive tables in Hadoop.

March 2019
 * Get users of Monolog based events to use new tables.
 * Begin migrating some Mediawiki 'Eventbus' eventgs to EventGate.
 * If possible decommission 'analytics' Kafka cluster.



=Q4 Goals =

Outcome X / Output X
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
 * Nullam interdum, elit in malesuada aliquam, libero lorem auctor lacus, eu mattis lacus velit vitae mauris.

Dependancies on: ___________

Goal(s)

 * Ut eget sodales odio. Maecenas a varius leo.

Status
April 2019
 * Discussed...

May 2019
 * Discussed...

June 2019
 * Discussed...