Production-like vagrant/Notes-2014-01-06
Idea: Make a vagrant image that mimics production as closely as possible and is portable (usb stick or other means)
- Ori says it's viable!
- Do we want to try and reproduce production cluster? This would be a huge resource drain for laptops etc, but perhaps could be more strategic about specific portions of the infra
Competing tensions
- Just a performant, easy to set up VM to get my development work done
- Make it as similar to production as possible (performance costs, probably can never go all the way on a laptop)
- Try cool new stuff like Redis as the only cache (instead of memcached)
Have a production role or just always do it?
- Ideally these would be set up using the same puppet code as production...which means more modules, more git submodules! :)
- Other opsen might be hesitant about this
- Ori might be hesitant about this :) it is possible and great but kinda hard sometimes!
(v = achievable at hackathon; ? = maybe?)
v * Varnish - Yuvi's patch https://gerrit.wikimedia.org/r/#/c/87623/ v * SSL termination w/Nginx v * MariaDB instead of MySQL v * Memcached / Redis switching ? * CentralAuth (nobody expects the CentralAuth) v * XML dump import (filtered, templates from commons, enwiki, etc); prepopuliating with specific content
-- there's a maintenance script importDump.php that could be automated from within puppet wikidata
- Full eventlogging setup
- logstash
Bringing production/vagrant puppet repos in line
- really difficult to do this well
- would require a lot of effort on the ops side
- Same problem between production and Labs; not the main goal but we should keep it in mind
- too much special casing in labs
- gotta put in the time and effort to do the abstraction - but usually stuff gets done in a hurry and doesn't go through the trouble; results in lots of conditionals
- would we get *that* much benefit from this approach for something like maraidb?
wmf-config repository does a lot of interesting stuff. Vagrant has a whole different mechanism
Consensus emerging around not necessarily totally bringing production/vagrant puppet inline with one another - perhaps target a few essential things (eg varnish, ssl), and deal wiht the rest as necessary
MediaWiki-Vagrant should be easy to distribute and set up at events
https://bugzilla.wikimedia.org/show_bug.cgi?id=49057
Create CentralAuth role https://bugzilla.wikimedia.org/show_bug.cgi?id=52306
Related idea: puppet/vagrant bootcamp
- Learn just enough puppet to be able to create a role that installs and configures an extension
- Hands-on workshop to add a set of new extensions to mw-vagrant
- Maybe all of the extensions used by commons?
Separate project:
Make Vagrant super easy to distribute Minimal spec for prep -
provide dedicated box for all the packages for use on the LAN provide a LiveCD
Participation? Ori: super busy, can advise, WILL PARTICIPATE will be at Zurich. Needs to run a performance workshop at some point tho. Matt: wants to help at hackathon, willing to help with pre setup stuff, but busy now Yuvi: can help with prepackaging stuff, willing to help out at Zurich Bryan: definitely wants to do this; start with bootcamp at Zurich Otto: along for the ride and willing to help out however
Add'l folks to reach out to: Erik B, Adam W, Moritz (math prof dude trying to fix all the math rendering stuff), Chris Steipp (for centralauth help), Alex (akosiaris)
Reasonable goal for the hackathon:
* Varnish would be the most useful thing we don't reproduce in vagrant * SSL termination * mariadb * Memcache/redis ... almost there already * CentralAuth? There be dragons. But hella important.
Goal: Focusing on a handful of high-priority puppet roles, get as close as we can to production-like vagrant instance
Next steps:
* Reaching out to additonal folks (including ops) * Work on distribuiton (Matt's going to spearhead this with Yuvi) ** Make proposal for hardware needs (usb stick, cds, blah) * Set up project proposal page for production-like vagrant (Arthur) * Set up project proposal page for bootcamp (Bryan) * Meet again in a month (Arthur) * Let Erik know we'll have hardware needs, get him specifics in a month (Arthur)