Wikimedia Product/Analytics Infrastructure/Workflow

References:

• What is an Event Schema and why do we need them?
• What is an Event Stream and why do we need them?
• Why can't I just send raw JSON?
• Why not develop directly on Beta?
• Can I skip the Beta step?
• Help, my application can't use the Beta cluster

Defs:

• Beta development environment: Applications that are not MediaWiki can have their own development environments or versions, and in these environments or versions, their application code will call APIs etc. belonging to the MediaWiki beta deployment, rather than the production deployment. When developing MediaWiki, it is right to say simply that you "deploy to the beta cluster", but individual applications outside of MediaWiki but which talk to MediaWiki, are more appropriately set up in their own way to talk to the beta cluster. TODO: Figure out better name and use this definition throughout.

Phase	Milestone	Event Schema	Event Stream	Instrumentation	Consult
Design	Question			Determine the set of metrics that could/should be gathered from application state to answer the question. See How to choose metrics.	Product Analytics
	Browse	Look for schemas you can re-use from the Event Schema Catalog.	Choose a name for your Event Stream by consulting the Event Stream Catalog.		Product Analytics
	Draft	Draft an Event Schema.	Draft an Event Stream Configuration.		Product Analytics
Phase	Milestone	Event Schema	Event Stream	Instrumentation	Consult
Local test	Checkout	Check out Event Schema Repository.	Check out mediawiki-config repository.	Check out software being instrumented.
	Edit	Edit Event Schema Repository to add new event schema.	Edit mediawiki-config repository add new event stream configuration.	Edit software being instrumented to add new instrument. See Event Platform Client Library#Adding_New_Instrumentation.
	Test			Wire Testing: Verify by inspection that events of the new kind are being produced over HTTP when you exercise the appropriate application components as designed. See Testing_Events#MediaWiki-Vagrant_Wire_Inspection.
Phase	Milestone	Event Schema	Event Stream	Instrumentation	Consult
Beta test	Submit	Submit edits to event schema repository.	Submit edits to mediawiki-config repository.	Submit edits to beta deployment process.
	+1	Changes pass CI.	Changes pass CI.	Changes pass CI.
	+2	Changes pass CR.	Changes pass CR.	Changes pass CR.
	Merge	Changes merged to beta. Automatic sync to production.	Changes merged to beta.	Changes merged to beta development environment.
	Test			Wire Testing: Verify by inspection that events of the new kind are being produced over HTTP when you exercise the appropriate application components as designed. See Testing_Events#Beta_Wire_Inspection.
	Test			Receipt Testing: Verify by inspection that events of the new kind are being received by the beta analytics server. See Testing_Events#Beta_Receipt_Inspection.
Phase	Milestone	Event Schema	Event Stream	Instrumentation	Consult
Limited production	Estimate		Prepare an estimate of throughput in events/unit. Edit the stream configuration to set the sample rate to a minimum value. See Event Platform Guidelines#Estimating Event Throughput and Event Platform Guidelines#Choosing a sampling rate.		Analytics Engineering, Product Analytics
	Submit		Commit and push your local changes to the mediawiki-config repository to the production repository.	Commit and push your local changes to your application code production repository.
	+1		Changes pass CI.	Changes pass CI.
	+2		Changes pass CR.	Changes pass CR.
	Merge		Changes merged to production.	Changes merged to production.
	Test			Wire Testing: Verify by inspection that events of the new kind are being produced over HTTP when you exercise the appropriate application components as designed. See Testing_Events#Beta_Wire_Inspection.
	Test			Receipt Testing: Verify by inspection that events of the new kind are being received by the analytics cluster. See Testing_Events#Production_Receipt_Inspection.
	Test			Log Monitoring: Monitor your application's production error logs for any problems caused by the new instrumentation.
Phase	Milestone	Event Schema	Event Stream	Instrumentation	Consult
Full production	Edit		Edit stream configuration to set the sample rate to desired value.		Analytics Engineering, Product Analytics
	Submit		Commit and push your local changes to the mediawiki-config repository to the production repository.
	+1		Changes pass CI.
	+2		Changes pass CR.
	Merge		Changes merged to production.
	Test			Log Monitoring: Monitor your application's production error logs for any problems caused by the new instrumentation. Monitor throughput on some dashboard.