Production-like vagrant/Notes-2014-01-06

Idea: Make a vagrant image that mimics production as closely as possible and is portable (usb stick or other means)
 * Ori says it's viable!
 * Do we want to try and reproduce production cluster? This would be a huge resource drain for laptops etc, but perhaps could be more strategic about specific portions of the infra

Competing tensions
 * Just a performant, easy to set up VM to get my development work done
 * Make it as similar to production as possible (performance costs, probably can never go all the way on a laptop)
 * Try cool new stuff like Redis as the only cache (instead of memcached)

Have a production role or just always do it?


 * Ideally these would be set up using the same puppet code as production...which means more modules, more git submodules! :)
 * Other opsen might be hesitant about this
 * Ori might be hesitant about this :) it is possible and great but kinda hard sometimes!

(v = achievable at hackathon; ? = maybe?)

v * Varnish - Yuvi's patch https://gerrit.wikimedia.org/r/#/c/87623/ v * SSL termination w/Nginx v * MariaDB instead of MySQL v * Memcached / Redis switching ? * CentralAuth (nobody expects the CentralAuth) v * XML dump import (filtered, templates from commons, enwiki, etc); prepopuliating with specific content -- there's a maintenance script importDump.php that could be automated from within puppet wikidata
 * Full eventlogging setup
 * logstash

Bringing production/vagrant puppet repos in line
 * really difficult to do this well
 * would require a lot of effort on the ops side
 * Same problem between production and Labs; not the main goal but we should keep it in mind
 * too much special casing in labs
 * gotta put in the time and effort to do the abstraction - but usually stuff gets done in a hurry and doesn't go through the trouble; results in lots of conditionals
 * would we get *that* much benefit from this approach for something like maraidb?

wmf-config repository does a lot of interesting stuff. Vagrant has a whole different mechanism

Consensus emerging around not necessarily totally bringing production/vagrant puppet inline with one another - perhaps target a few essential things (eg varnish, ssl), and deal wiht the rest as necessary

MediaWiki-Vagrant should be easy to distribute and set up at events https://bugzilla.wikimedia.org/show_bug.cgi?id=49057

Create CentralAuth role https://bugzilla.wikimedia.org/show_bug.cgi?id=52306

Related idea: puppet/vagrant bootcamp
 * Learn just enough puppet to be able to create a role that installs and configures an extension
 * Hands-on workshop to add a set of new extensions to mw-vagrant
 * Maybe all of the extensions used by commons?

Separate project: Make Vagrant super easy to distribute Minimal spec for prep - provide dedicated box for all the packages for use on the LAN provide a LiveCD

Participation? Ori: super busy, can advise, WILL PARTICIPATE will be at Zurich. Needs to run a performance workshop at some point tho. Matt: wants to help at hackathon, willing to help with pre setup stuff, but busy now Yuvi: can help with prepackaging stuff, willing to help out at Zurich Bryan: definitely wants to do this; start with bootcamp at Zurich Otto: along for the ride and willing to help out however

Add'l folks to reach out to: Erik B, Adam W, Moritz (math prof dude trying to fix all the math rendering stuff), Chris Steipp (for centralauth help), Alex (akosiaris)

Reasonable goal for the hackathon: * Varnish would be the most useful thing we don't reproduce in vagrant * SSL termination * mariadb * Memcache/redis ... almost there already * CentralAuth? There be dragons. But hella important.

Goal: Focusing on a handful of high-priority puppet roles, get as close as we can to production-like vagrant instance

Next steps: * Reaching out to additonal folks (including ops) * Work on distribuiton (Matt's going to spearhead this with Yuvi) ** Make proposal for hardware needs (usb stick, cds, blah) * Set up project proposal page for production-like vagrant (Arthur) * Set up project proposal page for bootcamp (Bryan) * Meet again in a month (Arthur) * Let Erik know we'll have hardware needs, get him specifics in a month (Arthur)