Extension:EventLogging/Todos

If you're interested in diving in, get in touch with Ori Livneh.

Server-side schema handling

 * Write Python abstraction for grabbing schemas from metawiki.
 * Validate incoming events against declared schema.
 * Generate SQL schema from JSON Schema (WIP: see 'glass' project in Gerrit).
 * Automatically INSERT TABLE when new schema encountered. (But carefully consider security and scalability implications.)

Monitoring

 * Watch for truncated events (tell-tale sign: missing trailing ';' in query string).
 * Keep sequence ID counters (one per host) and watch for gaps, which indicate packet loss.
 * Keep tabs on rate of incoming invalid events and emit alerts as appropriate.
 * Emit alerts as bona fide, subscribable events.
 * Write gmond plugin to send stats to Ganglia.
 * Create new $wgDebugLogGroup that writes to vanadium; use it to log EventLogging alerts from Apaches.

Storage / archiving

 * Set up automatic archiving and log rotation of raw event log data dump.
 * Figure out a sane MySQL permissions scheme.
 * Make sure Hadoop is getting all events, not just esams.
 * Make sure MySQL insert failures are handled gracefully.

Misc

 * Puppetize.
 * More unit tests.
 * Documentation.
 * DevServer.php should validate schema (WIP, staged in Ori's repo)
 * Improve dev tooling on Metawiki. Write a a small JavaScript module for Schema: pages that:
 * ) generates the $wgResourceLoaderModules declaration, so one can simply copy/paste schema module setup code.
 * ) provides a textarea for pasting a JSON object and checking if it validates against the schema.

- Test varnish patch referenced in RT 4094. Let Mark know how it goes.