Production-like vagrant/Notes-2014-01-06

From mediawiki.org

Idea: Make a vagrant image that mimics production as closely as possible and is portable (usb stick or other means)

  • Ori says it's viable!
  • Do we want to try and reproduce production cluster? This would be a huge resource drain for laptops etc, but perhaps could be more strategic about specific portions of the infra

Competing tensions

  • Just a performant, easy to set up VM to get my development work done
  • Make it as similar to production as possible (performance costs, probably can never go all the way on a laptop)
  • Try cool new stuff like Redis as the only cache (instead of memcached)

Have a production role or just always do it?

  • Ideally these would be set up using the same puppet code as production...which means more modules, more git submodules! :)
    • Other opsen might be hesitant about this
    • Ori might be hesitant about this :) it is possible and great but kinda hard sometimes!

(v = achievable at hackathon; ? = maybe?)

v * Varnish - Yuvi's patch https://gerrit.wikimedia.org/r/#/c/87623/ v * SSL termination w/Nginx v * MariaDB instead of MySQL v * Memcached / Redis switching ? * CentralAuth (nobody expects the CentralAuth) v * XML dump import (filtered, templates from commons, enwiki, etc); prepopuliating with specific content 

  -- there's a maintenance script importDump.php that could be automated from within puppet
 wikidata
  • Full eventlogging setup
  • logstash

Bringing production/vagrant puppet repos in line

  • really difficult to do this well
  • would require a lot of effort on the ops side
  • Same problem between production and Labs; not the main goal but we should keep it in mind
    • too much special casing in labs
  • gotta put in the time and effort to do the abstraction - but usually stuff gets done in a hurry and doesn't go through the trouble; results in lots of conditionals
  • would we get *that* much benefit from this approach for something like maraidb?

wmf-config repository does a lot of interesting stuff. Vagrant has a whole different mechanism


Consensus emerging around not necessarily totally bringing production/vagrant puppet inline with one another - perhaps target a few essential things (eg varnish, ssl), and deal wiht the rest as necessary


MediaWiki-Vagrant should be easy to distribute and set up at events https://bugzilla.wikimedia.org/show_bug.cgi?id=49057

Create CentralAuth role https://bugzilla.wikimedia.org/show_bug.cgi?id=52306

Related idea: puppet/vagrant bootcamp

  • Learn just enough puppet to be able to create a role that installs and configures an extension
  • Hands-on workshop to add a set of new extensions to mw-vagrant
    • Maybe all of the extensions used by commons?

Separate project: 

   Make Vagrant super easy to distribute
   Minimal spec for prep -

provide dedicated box for all the packages for use on the LAN provide a LiveCD

Participation? Ori: super busy, can advise, WILL PARTICIPATE will be at Zurich. Needs to run a performance workshop at some point tho. Matt: wants to help at hackathon, willing to help with pre setup stuff, but busy now Yuvi: can help with prepackaging stuff, willing to help out at Zurich Bryan: definitely wants to do this; start with bootcamp at Zurich Otto: along for the ride and willing to help out however

Add'l folks to reach out to: Erik B, Adam W, Moritz (math prof dude trying to fix all the math rendering stuff), Chris Steipp (for centralauth help), Alex (akosiaris)


Reasonable goal for the hackathon: 

   * Varnish would be the most useful thing we don't reproduce in vagrant
   * SSL termination
   * mariadb
   * Memcache/redis ... almost there already
   * CentralAuth? There be dragons. But hella important.

Goal: Focusing on a handful of high-priority puppet roles, get as close as we can to production-like vagrant instance

Next steps: 

   * Reaching out to additonal folks (including ops)
   * Work on distribuiton (Matt's going to spearhead this with Yuvi)
   ** Make proposal for hardware needs (usb stick, cds, blah)
   * Set up project proposal page for production-like vagrant (Arthur)
   * Set up project proposal page for bootcamp (Bryan)
   * Meet again in a month (Arthur)
   * Let Erik know we'll have hardware needs, get him specifics in a month (Arthur)
Retrieved from "https://www.mediawiki.org/w/index.php?title=Production-like_vagrant/Notes-2014-01-06&oldid=2545136"