Wikimedia Release Engineering Team/Checkin archive

This is the Wikimedia Release Engineering Team's archive of our weekly check-ins. We take notes on an etherpad during the meeting and archive them here afterwards. = 2015-01-20 =

Team Business

 * Team membership: members+=2 (hopefully)  (announcement pending)
 * Team responsibilities in the post "Mark & Markus" world
 * Quarterly Review run-through
 * https://docs.google.com/a/wikimedia.org/presentation/d/1vQBXZ-dNWn3RjSjERhiwEw4HHCW7_Z6l_zfBg1CIKHM/edit#slide=id.p

All Hands

 * https://etherpad.wikimedia.org/p/relengallhandsconversations

Vacations/Confs/etc
= 2015-01-13 = Guest: Antoine's daughter
 * Jan 20th - 27th: All Hands/Tech Days
 * Antoine flying on Jan 19th. Rest on Friday Jan 30th. Relocating (same building though) around Feb 14th.
 * Željko at FOSDEM conference (Brussels, Belgium) January 30-February 2
 * May: Hackathon in Lyon, France
 * Lyon used to be the capital of the area a long time ago. Nice old city, lot of great food, reasonably sunny/hot.

Team Business

 * Computer History Museum on Saturday the 24th, 9:30am pickup time from hotel probably
 * Zuul/Gerrit aren't talking again, needs a restart
 * Someone help Chris with his ssh config :)

Team Workboard
https://phabricator.wikimedia.org/project/board/20/?order=priority https://www.mediawiki.org/wiki/User:Cmcmahon/releng_process. Edit at will, and I'll show Arthur soon.
 * Met with Arthur about team practices/process stuff. Arthur suggest a "value stream map". I did this as at least a start:

Scrum of Scrums
https://phabricator.wikimedia.org/project/board/64/
 * Nothing outstanding

Vagrant
https://phabricator.wikimedia.org/project/view/627/ https://phabricator.wikimedia.org/project/board/627/
 * Helping to get ready for the MW-Vagrant Install Party
 * Establishing base unit-test coverage for plugin code ( https://phabricator.wikimedia.org/T76627 )

Browser tests
https://phabricator.wikimedia.org/project/view/630/
 * Collecting examples for Advanced Topics proposed session: https://phabricator.wikimedia.org/T86070 - https://www.mediawiki.org/wiki/MediaWiki_Developer_Summit_2015#Monday.2C_January_26
 * General session is: https://www.mediawiki.org/wiki/QA/Selenium_Workshop
 * Refactoring the page object for Flow
 * New TOC feature landed and only broke one thing, so I'll be picking up the pace here
 * Will probably report upstream bugs because of this
 * Need to update a VE test, I want to pair with Rummana on this if possible
 * Slowly hacking on MMV/UW tests, low priority
 * Want to start working with Dan's EAL when it's ready.
 * (It's merged and ready!)
 * Let's pair with feature devs to help educate
 * Antoine: congrats!

Deployment tooling
https://phabricator.wikimedia.org/project/board/349/query/open/?order=priority
 * https://phabricator.wikimedia.org/T85008 - Determine Trebuchet/git-deploy maintenance plan

Continuous Integration
https://phabricator.wikimedia.org/tag/continuous-integration/board/?order=priority Zuul deadlocks from time to time due to Gerrit not answering when Zuul report a comment and Zuul has no timeout
 * https://wikitech.wikimedia.org/wiki/Incident_documentation/20150106-Zuul
 * That is a rather new issue, with root cause being Gerrit failling to reach its database
 * Draft architecture to have Jenkins jobs in dispoable/isolated VMs: https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Isolation
 * multiple extensions tested together

Beta cluster
https://phabricator.wikimedia.org/project/board/497/?order=priority
 * YuviPanda has been very active:
 * removed almost all local hacks
 * refactored a few puppet class
 * moved some settings from operations/puppet.git to hiera ( https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep )
 * Labs DNS issue fixed (all DNS entries had a 0 TTL disabling DNS caching entirely => mass queries => server overloaded => DNS failing).
 * text / mobile caches transiently broken last week (vagrant issue)

Vacations/Confs/etc

 * Željko at Test Automation Bazaar conference (Austin, Texas) January 14-19
 * Jan 19th: MLK Day (US)
 * Jan 20th - 27th: All Hands/Tech Days
 * Antoine flying on Jan 19th. Rest on Friday Jan 30th. Relocating (same building though) around Feb 14th.
 * Željko at FOSDEM conference (Brussels, Belgium) January 30-February 2
 * May: Hackathon in Lyon, France
 * Lyon used to be the capital of the area a long time ago. Nice old city, lot of great food, reasonably sunny/hot.

= 2015-01-06 =

Team Business

 * Welcome back :)
 * Quarterly Review - Jan 23rd

Team Workboard
https://phabricator.wikimedia.org/project/board/20/?order=priority

Scrum of Scrums
https://phabricator.wikimedia.org/project/board/64/
 * Nothing much. Got Flow folk to review and merge two patches after the last SoS

Vagrant
https://phabricator.wikimedia.org/project/view/627/

Browser tests
https://phabricator.wikimedia.org/project/view/630/
 * Asked to refactor a Flow test https://phabricator.wikimedia.org/T85201. Unfortunately or not, doing this properly is going to require something clever. I'm spiking that right now, and will ask Dan for some review before I get too far down the road. https://gerrit.wikimedia.org/r/#/c/182851/
 * Still pairing with Rummana on VE stuff
 * Would like to start training Elena as well.
 * Updated a bunch of docs on mw.o
 * Apologia for spending the end of 2014 refactoring: https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Refactoring_2014.

Deployment tooling
https://phabricator.wikimedia.org/project/board/349/query/open/?order=priority

Continuous Integration
https://phabricator.wikimedia.org/tag/continuous-integration/board/?order=priority Zuul/Jenkins/Gearman/Gerrit yesterday Figure out CI stuff for Phabricator: https://phabricator.wikimedia.org/T85123
 * MediaWiki tests now run under both HHVM and Zend!
 * Started using composer as an entry point to run tests under Jenkins. Example repos: cdb mediawiki/tools/codesniffer
 * Basic convention for test entry points https://www.mediawiki.org/wiki/Continuous_integration/Tests_entry_points
 * The ruby section could use to be filled
 * HHVM now autoupgrade on CI Jenkins slaves
 * testing mobile related mw extensions together: done to one last bug in Echo https://phabricator.wikimedia.org/T78592
 * Run Jenkins jobs in isolated VMs. Antoine will reorganize / break up the mess of tasks under https://phabricator.wikimedia.org/T47499
 * https://wikitech.wikimedia.org/wiki/Incident_documentation/20150106-Zuul
 * TL:DR; If Gerrit is unavailable when Zuul send a comment, Zuul deadlocks => restart Zuul
 * https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart, need shell access
 * make sure people have access to gallium (Chris M, Dan, Mukunda, Zeljko)
 * https://phabricator.wikimedia.org/T85936
 * Example: https://gerrit.wikimedia.org/r/#/c/181211/ . Need ops 3 days retention + manager approval. Probably some training by Antoine/Timo in addition to the Zuul doc above.
 * Zuul cloner does not fetch submodules nor does it clean the workspace. Cause a few oddities. Antoine has Zero bandwith to implements the features.
 * VisualEditor-npm is impacted

Beta cluster
https://phabricator.wikimedia.org/project/board/497/?order=priority ATDD: Acceptance Test Driven Development https://en.wikipedia.org/wiki/Acceptance_test-driven_development
 * wmflabs DNS has been flappy for 2+ weeks causing the update jobs and puppet run to randomly fails. Lovely spam from Shinken and on qa-alerts
 * Please ping Coren :-]
 * sudo access on mobile-cache instance is back
 * Q3 "intra-departamental priority"
 * docs/plan-to-be: https://www.mediawiki.org/wiki/Beta_cluster/2014-15-Q3
 * is Q3 july-september? :D
 * Talk to Arthur about a TP+QA joint production. Can we do ATDD? Anywhere?
 * Would like to schedule a session for Tech Days. I like "Browser Test Design": ATDD and intent; waiting; proper code re-use; page object design; nesting elements.https://www.mediawiki.org/wiki/MediaWiki_Developer_Summit_2015#How_to_schedule_a_session

Hiring

 * Release Engineer in-progress (some in-person interviews next week)

Vacations/Confs/etc

 * Jan 19th: MLK Day (US)
 * Jan 20th - 27th: All Hands/Tech Days
 * Antoine flying on Jan 19th. Rest on Friday Jan 30th
 * May: Hackathon in Lyon, France
 * Lyon used to be the capital of the area a long time ago. Nice old city, lot of great food, reasonably sunny/hot.

= 2014-12-16 =

Team Workboard
https://phabricator.wikimedia.org/project/board/20/?order=priority

Scrum of Scrums
https://phabricator.wikimedia.org/project/board/64/

Vagrant
https://phabricator.wikimedia.org/project/view/627/
 * Triage existing tasks
 * Create workboard?
 * Bigger goal of establishing unit-test coverage

Browser tests
https://phabricator.wikimedia.org/project/view/630/
 * Finalizing environment abstraction layer
 * Further testing by refactoring some advanced scenarios (in Flow, CentralAuth, Echo)
 * Team review tomorrow
 * Merging later this week
 * all VE tests are broken. I'll walk Rummana through fixing them later today.
 * Writing a new test for vFlow since they marked the ticket High and haven't done it themselves. It'll be fast. https://phabricator.wikimedia.org/T75239
 * Refactoring MMV and UW tests as their UIs get the OOJS treatment, working mostly with Gilles and a little with Mark. Trying to get rid of the sleeps and I'm finding some issues with how Chrome treats the z-axis in our UIs. I want to pair with Zeljko on this later.

Deployment tooling
https://phabricator.wikimedia.org/project/board/349/query/open/?order=priority
 * Mukunda submitted patch (https://gerrit.wikimedia.org/r/#/c/180200/) for https://phabricator.wikimedia.org/T72054
 * [scap] Syncing wmf-config/PrivateSettings.php syncs symlink and not file contents

Continuous Integration
https://phabricator.wikimedia.org/tag/continuous-integration/board/?order=priority
 * Antoine getting HHVM job
 * manual run of them all, assess status of the builds, if only a few broken, enable it everywhere
 * Gearman/Zuul/Jenkins still deadlocking from time to time for random reasons :-\

Beta cluster
https://phabricator.wikimedia.org/project/board/497/?order=priority We have new labs servers! If you collect a rough count of how many new instances you need, I'll lift the quotas.

Vacations/Confs/Holidays/etc

 * 12/10-12/17 - Elena's on vacation
 * 12/20 - 1/4 - Greg on vacation
 * 12-22 to 1-06 Željko vacation
 * 12/24 - Chris vacation
 * 12/25-1/4 - Dan vacation
 * 12/25 - Christmas
 * 12/26 - Day after Christmas
 * 1/1 - New Years
 * 1/2 - Day after New Years
 * Antoine has no clue yet. Will be on call at worse.

= 2014-12-09 =

Team Workboard
https://phabricator.wikimedia.org/project/board/20/?order=priority

Scrum of Scrums
https://phabricator.wikimedia.org/maniphest/query/iYosD4_9SSPW/#R

Vagrant
https://phabricator.wikimedia.org/project/view/627/
 * Dan to email out the survey results to wikitech-l
 * next priority: unit test suite for MW-V

Browser tests
https://phabricator.wikimedia.org/project/view/630/ Zeljko and I discussed making browser tests voting. * Would require a mediawiki instance capable of pulling a branch upstream from master *and* have Vagrant-style roles available. ** This might require a full-fledged Vagrant running on the instance (we'd need an LXC or Docker provider)

Deployment tooling
https://phabricator.wikimedia.org/project/board/349/query/open/?order=priority

Continuous Integration
https://phabricator.wikimedia.org/tag/continuous-integration/board/?order=priority
 * Slaves running hhvm - https://phabricator.wikimedia.org/T75356
 * Ori/Guiseppe offered help, what's next here?
 * Antoine lacks brain cycles / afraid of context switch from (python,ci) to (hhvm,puppet)
 * Going to tests mobile related extensions together (Mantle, MobileFrontend ... ). Poked mobile team about it: https://lists.wikimedia.org/pipermail/mobile-l/2014-December/008398.html

Beta cluster
https://phabricator.wikimedia.org/project/board/497/?order=priority
 * New virt hardware is in and racked!
 * We turn out to not know how to work these new HP boxes though, so there's lots of give-and-take with datacenter folks.
 * I'll be able to lift quota restricitions just as soon as those boxes are up, which could happen at any minute :/
 * antoine: Could use a quota raise on integration project as well (Jenkins slaves)
 * andrew: yeah doable once we get more hardware (see above)
 * antoine: could use instances teamplates (more core but less mem)
 * andrew: doable, fill a task :)
 * antoine: what about specific virtual images
 * andrew: it is not automatized and adds burden to the OpenStack infrastructure. It is prefered to use base image + puppet.

Process discussion
Thinking about process and (possibly) metrics. THIS IS A STRAW MAN BRAIN DUMP DRAFT

TL;DR RelEng team process and metrics should be on an axis of cooperation vs collaboration.


 * Collaboration is working with others to do a task and to achieve shared goals.
 * Cooperation is the process of groups of organisms working or acting together for their common/mutual benefit, as opposed to working in competition for selfish benefit.

(from Wikipedia)

Describing process and metrics for the ReEng team is challenging because of a) the diversity of individual activities and b) the nature of a support organization. Here is something we could say since we are a support organization: Counterexample: Goals: In the next time period: Metrics:
 * Apropos of a):
 * How is what Chris does similar to what Mukunda does similar to what Elena does, etc.?
 * Apropos of b)
 * If we work with the VE team, we use their process and conventions, with the Mobile team, their process, with Ops, their process, etc. etc.
 * We emphasize cooperation over competition. We want to cooperate with others in a useful way.
 * Example of increasing cooperation: not long ago only Gilles had contributed test code to the MultimediaViewer repo. Recently Chris has also begun contributing to MMV.
 * We emphasize collaboration over individual activity. We want to share our areas of expertise with others.
 * Example: Recently only Chris had contributed browser test code to the VisualEditor repo. Now Rummana has also begun contributing.
 * Bonus round: we create new ways to cooperate and collaborate
 * Example: Phabricator of course
 * Jenkins Job Builder
 * Moving from Cloudbees
 * Upcoming: e.g. make browser tests voting, needs collaboration and cooperation to make it happen
 * If you are always working alone, you are failing at collaboration.
 * If someone else is doing work that you could help with, you are failing at cooperation
 * Identify at least X number of projects to which you would contribute. (Cooperation)
 * Identify at least X number of projects projects with which you would share what you know. (Collaboration)
 * Since the last time noted:
 * Where have you contributed to a project or area new to you? (Cooperation)
 * How have you helped others understand the work you do? (Collaboration)

To increase Collaboration:
 * have both Rummana and Elena familiar with Jenkins

Vacations/Confs/Holidays/etc

 * 12/10-12/17 - Elena's on vacation
 * 12/15-16 - Greg in Michigan (mostly out on Monday, somewhat out on Tuesday, will really really really try to be here for the meeting)
 * 12/20 - 1/4 - Greg on vacation
 * 12-22 to 1-06 Željko vacation
 * 12/24 - Chris vacation
 * 12/25 - 1/3 - Andrew B on vacation
 * 12/25-1/4 - Dan vacation
 * 12/25 - Christmas
 * 12/26 - Day after Christmas
 * 1/1 - New Years
 * 1/2 - Day after New Years
 * Antoine has no clue yet. Will be on call at worse.

= 2014-12-02 =

Team Workboard
https://phabricator.wikimedia.org/project/board/20/?order=priority
 * (Antoine) should we get quarterly milestones tags? https://phabricator.wikimedia.org/T75729
 * Background idea: replace the wiki page, could be done some other way.

Scrum of Scrums
https://phabricator.wikimedia.org/maniphest/query/iYosD4_9SSPW/#R Chris will emphasize:
 * https://phabricator.wikimedia.org/T76086 needs Ops help
 * https://phabricator.wikimedia.org/T65681 can Aaron close this issue? We don’t know…
 * https://phabricator.wikimedia.org/T67683 needs Brandon Black’s help. He might not be aware of it.

Vagrant
https://phabricator.wikimedia.org/project/view/627/
 * Writing short survey summary; will send out today/tomorrow
 * Presentation went well (agreed :) )

Browser tests
https://phabricator.wikimedia.org/project/view/630/
 * Refactoring/untangling MobileFrontend browser test repo (updating to RSpec3 along the way)
 * Making rubocop fixes
 * Training Rummana (ongoing)
 * 2x/week pending interruptions
 * Considering a tech talk. http://www.mediawiki.org/wiki/Project:Calendar/How_to_schedule_an_event/TechTalks (all this refactoring makes me want to do a presentation about "Do's and Dont's for Browser Tests")
 * Finishing up Env Abs Layer
 * Patch to avoid hitting rubygems: https://gerrit.wikimedia.org/r/#/c/176818/
 * We keep hitting rubygems, which suffered from a DDoS on Monday thus causing browser tests to fail early
 * Discussion about providing bundler via Debian package (Dan, Antoine). TLDR: we gave up attempting to backport it
 * VisualEditor could use a few more tests

Deployment tooling
https://phabricator.wikimedia.org/project/board/349/query/open/?order=priority

Continuous Integration
https://phabricator.wikimedia.org/tag/continuous-integration/board/?order=priority
 * RFC approved! https://www.mediawiki.org/wiki/RFC/Extensions_continuous_integration
 * ACTION Antoine to fill related tasks and find a set of repos to play with
 * they don't want the changes to take longer than 5ish minutes
 * Zuul cloner unable to fetch tag additions https://phabricator.wikimedia.org/T76003
 * Mark B proposed to reuse old app servers for more Jenkins slaves
 * Set up HHVM on Jenkins slave, see ops list and https://phabricator.wikimedia.org/T75356

Beta cluster
https://phabricator.wikimedia.org/project/board/497/?order=priority
 * Missing udplog being worked on / fixed?
 * https://phabricator.wikimedia.org/T1295 "Make ::mediawiki::syslog and ::mediawiki::php logging destinations configurable via hiera"
 * https://phabricator.wikimedia.org/T74275 "no log in deployment-bastion:/data/project/logs from "503 server unavailable" on beta labs"

Vacations/Confs/Holidays/etc

 * 12/1 - 2 - Mukunda visiting parents (Can check IRC but on slow connection)
 * 12/10-12/17 - Elena's on vacation
 * 12/15-16 - Greg in Michigan (mostly out on Monday, somewhat out on Tuesday, will really really really try to be there for the meeting)
 * 12/20 - 1/4 - Greg on vacation
 * 12/24 - Chris vacation
 * 12/25 - 1/3 - Andrew B on vacation
 * 12/25-1/4 - Dan vacation
 * 12/25 - Christmas
 * 12/26 - Day after Christmas
 * 1/1 - New Years
 * 1/2 - Day after New Years
 * Antoine has no clue yet. Will be on call at worse.

= 2014-11-25 =

Team Business

 * Holidays
 * Andrew B changing to UTC+8 2014-12-6 - 2015-01-12
 * Andrew B vacation (afk) December 25th - January 3


 * Delegation planning
 * Deploys
 * Team
 * Phabricator!!!!!
 * Team board: https://phabricator.wikimedia.org/project/board/20/
 * Beta Cluster: https://phabricator.wikimedia.org/project/board/497/
 * Other projects don't have boards yet

Scrum of Scrums
(kept as long url because all short url providers are blocked by mw.org)
 * Dependency wall
 * All very quiet
 * Will migrate to Phabricator:
 * Page: https://phabricator.wikimedia.org/tag/scrum-of-scrums/
 * Has links to deps reports by Bryan Davis: https://phabricator.wikimedia.org/maniphest/query/iYosD4_9SSPW/#R

Phabricator

 * Antoine: it is awesome
 * OMG lots of work ;)
 * We made hacker news front page with this linkbait: https://blog.wikimedia.org/2014/11/24/welcome-to-phabricator-wikimedias-new-collaboration-platform/
 * https://news.ycombinator.com/item?id=8654403
 * Example dashboard https://phabricator.wikimedia.org/dashboard/view/6/
 * You can create your own dashboard
 * Dashboards can be "installed" to replace the default homepage
 * Well done Mukunda & all!

Vagrant

 * Tech talk, today! (in one hour)
 * EVERYONE ATTEND! :)
 * Finishing up survey analysis
 * Will send out findings

Jenkins

 * RFC https://www.mediawiki.org/wiki/RFC/Extensions_continuous_integration
 * discussed tomorrow Wed. 26th @ 21:00 UTC #wikimedia-office
 * simplified JJB doc templates to be usable on labs and recognize tags/branch
 * https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation
 * doc jobs being moved to labs slaves
 * work in progress to use composer as an entry point for Jenkins jobs (much like npm test)
 * QUnit migration to Karma
 * main runners from prod to labs
 * npm/Grunt that drives chromium to run qunit tests
 * They dont navigate like browser tests do!

Browser tests
All repos need mediawiki_selenium 0.4.1 and the Jenkins jobs HEADLESS_DESTROY_AT_EXIT=false. That is to fix a race condition between jobs that kill other jobs xvfb. "mediawiki_selenium always use the same default xvfb display 99" https://phabricator.wikimedia.org/T73602 which blocks: "Jenkins: browser test host performance issue for timed builds" https://phabricator.wikimedia.org/T68449
 * starting to refactor MobileFrontend tests
 * pairing with Rummana on new VisualEditor tests
 * some big changes in Flow function coming any minute, looking to minimize tech debt as that happens
 * builds are very green (pending known bugs)
 * any info on this one? https://phabricator.wikimedia.org/T75229
 * Environment abstraction layer is ready to showcase, iron out final bugs
 * Next week probably, will show to Chris/Željko/Antoine(?)
 * New exceptions added to Ruby coding conventions
 * https://www.mediawiki.org/wiki/Manual:Coding_conventions/Ruby

Beta cluster
https://phabricator.wikimedia.org/tag/beta-cluster/board/
 * Pick tasks!
 * Bunch of puppet failures since last week:
 * Antoine screwed puppet certs for half a day while attempting to fix an issue
 * manifests errors filled as Tasks by YuviPanda
 * Q: how do we get puppet gurus (read: ops) to fix them?
 * A: On our side to figure out a process to work with ops without being a pain in the a** for them :D

Deployment tooling
https://phabricator.wikimedia.org/maniphest/?statuses=open%2Cstalled&allProjects=PHID-PROJ-2fqz5e6sn7ysjcako5mj#R
 * All quiet Oh no reedy:
 * https://phabricator.wikimedia.org/T69931

Hiring
NOOP

Other
Antoine Q: how to propose a task for Scrum of Scrum Chris A: just add #scrum-of-scrum on the task and ping Chris McMahon with details about it.

Vacations/Confs/Holidays/etc

 * 11/25 - Željko on a local meetup during releng weekly meeting
 * 11/27 - Thanksgiving (US)
 * 11/28 - Day after Thanksgiving (US)
 * 12/1 - 2 - Mukunda visiting parents (Can check IRC but on slow connection)
 * 12/15-16 - Greg in Michigan (mostly out on Monday, somewhat out on Tuesday, will really really really try to be there for the meeting)
 * 12/20 - 1/4 - Greg on vacation
 * 12/25 - 1/3 - Andrew B on vacation
 * 12/25 - Christmas
 * 12/26 - Day after Christmas
 * 12/19-12/31 - Dan vacation
 * 1/1 - New Years
 * 1/2 - Day after New Years
 * Antoine has no clue yet. Will be on call at worse.

= 2014-11-18 =

Team Business

 * Team Health Survey in a few hours
 * https://www.mediawiki.org/wiki/Team_Practices_Group/Health_check_survey#Survey_focus_areas

Scrum of Scrums
(kept as long url because all short url providers are blocked by mw.org)
 * Dependency wall

Deployment tooling

 * All quiet

Jenkins

 * Upgraded to latest LTS
 * New UI!! https://integration.wikimedia.org/ci/
 * Jenkins was speaking Italian until Chad and Andrew fussed with it yesterday.  For some reason en_US = english but en = Italian
 * ACTION (antoine): fill a bug to upstream
 * Slow browser tests dashboard https://bugzilla.wikimedia.org/show_bug.cgi?id=70671
 * Antoine proposed to delete history on qa list https://lists.wikimedia.org/pipermail/qa/2014-October/002043.html
 * Could archive them via a cron
 * Greg: Or: "Jenkinks UI slow due to constant build record loading"https://issues.jenkins-ci.org/browse/JENKINS-15858 (Fixed in 2.8)
 * greg is wrong
 * Doc jobs being migrated out of gallium to Trusty slaves
 * MediaWiki core is using a new version of doxygen (1.8.x)
 * Mantle removed code which ended up breaking a few extensions on beta cluster. Would have been caught by running tests of all extensions together which is exactly https://www.mediawiki.org/wiki/RFC/Extensions_continuous_integration   Please have a read :-)

Beta cluster

 * Much healthier since we stopped those scap/HHVM 503 problems
 * Still have a few issues for MMV and UW, working the bugzilla tickets for them
 * Shinken: http://shinken.wmflabs.org/problems
 * lots of puppet issues
 * YuviPanda (now ops) doing a lot on monitoring front
 * We need tests for puppet :D

Vagrant

 * Survey analysis
 * base resources requirements have grown a lot
 * Tech talk prep

Browser tests
Theme is "address the tech debt in the browser test repos including rubocop"
 * Echo and Flow repos cleaned and refactored
 * VisualEditor repo almost finished with cleanup
 * MobileFrontend is next
 * TODO: bump mediawiki_selenium to 0.4.1+ on all repo
 * will let fix a race condition between jobs by setting: export HEADLESS_DESTROY_AT_EXIT=false
 * https://bugzilla.wikimedia.org/show_bug.cgi?id=71602#c6
 * Ruby files are ruby 2.0 so use the ruby2.0lint Jenkins job template on repos (not the 1.9.3)
 * Environment abstraction layer
 * Started high-level documentation (https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Environment_abstraction_layer)
 * Just a couple of missing pieces in Sauce integration

Hiring

 * RelEng still going...

Vacations/Confs/Holidays/etc

 * 11/25 - Željko on a local meetup during releng weekly meeting
 * 11/27 - Thanksgiving (US)
 * 12/15-16 - Greg in Michigan (mostly out on Monday, somewhat out on Tuesday, will really really really try to be there for the meeting)
 * 11/28 - Day after Thanksgiving (US)
 * 12/25 - Christmas
 * 12/26 - Day after Christmas
 * 1/1 - New Years
 * 1/2 - Day after New Years
 * 12/20 - 1/4 - Greg on vacation
 * 12/25-1/3 - Andrew B on vacation

=2014-11-04=

Team Business

 * No meeting next week (govt holidays)

Scrum of Scrums
(kept as long url because all short url providers are blocked by mw.org)
 * Dependency wall

Phabricator

 * Security and Herald - two herald rules (one admin one that sets security first, second one runs last to sanitize)
 * phabricator is spamming wikibugs-l, trying to deal with that somehow

Deployment tooling

 * Sean merged the l10n change (!!)
 * l10nupdate still disabled

Jenkins

 * RFC draft "Multiple repositories tested together" published https://www.mediawiki.org/wiki/Requests_for_comment/Extensions_continuous_integration (Antoine)
 * ACTION please please give it a look and give some feedback :D
 * One Zuul patch deployed, still pending upstream reviews (Antoine)
 * Antoine busy at OpenStack summit the whole week
 * Timo moving more jobs from prod slave to labs slave running Trusty (Timo)
 * https://phabricator.wikimedia.org/T1074
 * (Zeljkof) Rubocop done, announcement has been sent to wikitech list last week

Beta cluster

 * Monitoring effort going on:
 * Low level graphs of instances (YuviPanda, Timo) https://tools.wmflabs.org/nagf/?project=deployment-prep https://tools.wmflabs.org/nagf/?project=integration
 * Icinga production monitoring being migrated to Shinken, to be adjusted for beta cluster context (YuviPanda)
 * https://www.mediawiki.org/wiki/Wikimedia_Engineering/2014-15_Goals/Q3#Beta_Cluster
 * to be worked in collaboration with Ops and mediawiki-core teams
 * TODO/TO-ADD: Determine who uses Beta Cluster and why. Maybe a survey like the MediaWiki/Vagrant one

Browser tests

 * Major refactor for VisualEditor tests: no more iframes. Page Object rawks.
 * More pairing with Rummana coming
 * Many Flow updates, a few more to go
 * Builds turned green today: Core, UploadWizard, and pending BZ fixes VisualEditor and MobileFrontend
 * Testing out experimental env-abstraction-layer branch of MWS
 * Finishing RuboCop setup (Željko)
 * Željko looking at browsertests/Jenkins performance issue

Vagrant

 * (Dan) Work on November presentation w/ Bryan Davis
 * (Dan) Sending mediawiki-vagrant survey to wikitech-l

Hiring

 * RelEng in-progress

Vacations/Confs/etc

 * 11/3 - 11/7: Antoine - OpenStack Summit Paris
 * 11/11 - Antoine Holiday (WW1)
 * 11/11 - Veteran's Day (US)
 * 11/27 - Thanksgiving (US)

=2014-10-28=

Scrum of Scrums
(kept as long url because all short url providers are blocked by mw.org)
 * Dependency wall

Phabricator

 * test instance up at http://bugzillapreview.wmflabs.org/

Deployment tooling

 * 5am deploy window on Tuesday
 * Thurs -> Wed window move

Jenkins/Zuul

 * Jobs not being triggered by Zuul suddenly https://bugzilla.wikimedia.org/show_bug.cgi?id=63760, hard to track. Antoine attempting to reproduce reliably.
 * Antoine attending OpenStack summit next week and will meet upstream. Good opportunity to fix it for real :-)
 * Took random notes about a uniq job to test all extensions together. More later.

Beta cluster

 * scap broken because WikiGrok extension was not registered in mediawiki/extensions.git
 * https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment#Deploy_to_beta_cluster_on_Labs
 * (ACTION Antoine to update the Wiki pages)

Browser tests

 * (Dan) Env abstraction layer thingy ready for some manual/functional testing
 * start with CentralAuth
 * experimental branch with multiple commits
 * (Dan) Sorting through rubocop-related commits and rewriting Ruby conventions

Vagrant

 * (Dan) Sent a survey reminder out (we're up to 49 responses (!!!))
 * next step: sending to wikitech-l/etc
 * (Dan) Working with Bryan on an outline for the tech talk in November

Hiring

 * RelEng in-progress

Vacations/Confs/etc

 * 10/20 - 10/30 - Antoine might skip morning and work during evening {european pov}.
 * 10/30 - 10/31 Chris vacation
 * 11/3 - 11/7: Antoine - OpenStack Summit Paris
 * 11/11 - Antoine Holiday (WW1)
 * 11/ something Thanksgiving (US)

=2014-10-21=

Team Business

 * Phabricator workboard discussion
 * https://phabricator.wikimedia.org/project/board/20/
 * Make sure you have an account :)
 * Join the "project"
 * Things to think about, we probably want to take a 2 hour chunk of time to do this well enough:
 * What is our generalized process? We can keep our current process and make the tool match that.
 * "size" of things in this specific board (do we need two?)
 * "Blocked" column?
 * "Needs review" column?
 * "Done" column (and related "Archive" column)?
 * other things?

Scrum of Scrums
(kept as long url because all short url providers are blocked by mw.org)
 * Dependency wall

Phabricator

 * Direct phabricator / migration questions to the #wikimedia-devtools irc channel
 * QChris finished the phabricator plugin for gerrit, we should see gerritbot posting to phabricator soon
 * https://phabricator.wikimedia.org/T169
 * See a sample of data migrated from bugzilla to phabricator to get a feel for how it's going to look and identify any problems:
 * Preview instance: https://bugzillapreview.wmflabs.org
 * (antoine) it (phabricator upstream:) is awesome. Can fill bug at https://secure.phabricator.com/ which are triaged quickly

Deployment tooling

 * ready to merge the l18n stuff
 * sync up with Niklas re l18n

Jenkins
Zuul / Gearman related:
 * Jobs being stuck in Zuul queue due to an error not being handled https://bugzilla.wikimedia.org/show_bug.cgi?id=72113, patch proposed upstream
 * Jobs not being triggered by Zuul suddenly https://bugzilla.wikimedia.org/show_bug.cgi?id=63760, hard to track
 * Fixed: "Jenkins: jobs created via JJB are not properly registered in Zuul Gearman server" https://bugzilla.wikimedia.org/show_bug.cgi?id=63758, pending upstream release although already deployed
 * (Zeljkof, Antoine) some rubocop ruby2.0 related work over the week.

Beta cluster

 * Have some Icinga/Graphite monitoring in staff. Not sure whom to notify beside Yuvi, Greg and I. Ideas? Maybe a public mailling list similar to qa-alerts ( https://lists.wikimedia.org/pipermail/qa-alerts/ ), but not qa-alerts since it is super spammy. Maybe betacluster-alerts or something
 * create a task in the workboard (greg)
 * Andrew is still waiting on bids for virt hardware, and frowning a lot

Browser tests

 * (Dan) Making progress on env layer
 * Antoine got Dan to commit to giving a TDD talk to WMF :)
 * Chris in SF, working mostly directly with Flow folks (mostly S I think) and a little with Rummana for VE.
 * Almost 80 people RSVPd for http://www.meetup.com/wikimedia-tech/events/207856222/

Vagrant

 * (Dan) Survey out—around 38 replies so far. Let it roll? Send to community?
 * (Dan) Start analysis of survey results

Hiring

 * Release Engineer in-progress: https://boards.greenhouse.io/wikimedia/jobs/29435?t=5fw24x

Vacations/Confs/etc

 * 10/20 - 10/30 - Antoine might skip morning and work during evening {european pov}.
 * 10/21-22-23 Chris in SF (Elisabeth Hendrickson talk at WMF Oct 22)
 * 10/23 - Antoine traveilling during morning
 * 10/27 - 10/29 Chris at Google Test Automation Conference Seattle
 * 10/30 - 10/31 Chris vacation
 * 11/3 - 11/7: Antoine - OpenStack Summit Paris
 * 11/11 - Antoine Holiday (WW1)

=2014-10-14=

Scrum of Scrums

 * Dependency wall: https://wikimedia.mingle.thoughtworks.com/projects/scrum_of_scrums/cards/list?style=list&tab=All

Jenkins

 * Jobs created with Jenkins Job Builder should now properly register ( https://bugzilla.wikimedia.org/show_bug.cgi?id=63758 )
 * Tobi completed WD migrated (refractor in-progress)
 * rubucop etc
 * "Jenkins Performance"
 * No remaining time last to reproduce it

Beta cluster

 * virt100x outage last week, we have lost deployment-cxserver01 . Had trouble rebuilding it due to labs partitionning improvement. Kindly fixed and enhanced by Andrew/Coren
 * Andrew and Rob reviewing quotes for additional virt hardware today (re: new beta cluster)
 * Maybe we need to backup instances
 * Second beta cluster still being discussed

Browser tests

 * Stuff went red while Chris was away
 * mediawiki/selenium has env to fix xvfb race condition. Only used for local browser tests though.
 * since all jobs are on Sauce, should we stop throttling them?
 * Let Chris work on getting the builds back to green first
 * Jobs were running SauceLabs and had timeouts, so unrelated to xvfb race condition
 * Conclusion: keep them throttle-
 * Continuing work on environment abstraction

Vagrant

 * Release trafficcontrol MWV role (uses tc + iptables to simulate network conditions)
 * Perform MMV perf tests using trafficcontrol profiles
 * Distribute survey! (today)

Hiring

 * Elena starts today(!!)
 * Buddying with James and Rummana, I chatted with her yesterday, will sync up later in the week as well
 * Chris wrote this long ago but it's still mostly relevant: https://www.mediawiki.org/wiki/Quality_Assurance/first_week
 * welcome email to-be-sent (already written)

Vacations/Confs/etc

 * 10/13 - 10/17 Greg at MediaWiki Core offsite
 * 10/20 - 10/30 - Antoine might skip morning and work during evening {european pov}.
 * 10/21-22-23 Chris in SF (Elisabeth Hendrickson talk at WMF Oct 22)
 * 10/27 - 10/29 Chris at Google Test Automation Conference Seattle
 * 10/30 - 10/31 Chris vacation
 * 11/3 - 11/7: Antoine - OpenStack Summit Paris
 * 11/11 - Antoine Holiday (WW1)

= 2014-10-07 =

Team Business

 * All Hands Excursion
 * https://docs.google.com/a/wikimedia.org/spreadsheets/d/1qTLhQstAiTIHH6iBjHINLuaOaZHs1CnIAYWmcLjxQCM/edit#gid=0

Scrum of Scrums
One card (ops/release) Phabricator related:
 * Dependency wall: (grrr spam filter)
 * https://wikimedia.mingle.thoughtworks.com/projects/scrum_of_scrums/cards/119

Phabricator
Lots happening in Phabricator world.....
 * We announced that the wikimedia phabricator is now open to everyone...
 * log in (or register) and try to break it
 * link your account to LDAP and OAuth provider here: https://phabricator.wikimedia.org/settings/panel/external/
 * New projects opening on hold till Bugzilla migration
 * bug qgil if you have an urgent need for a new project before the migration
 * Lots of tasks finally closed out of spite.  https://phabricator.wikimedia.org/maniphest/query/oRIJDB5MpxjI/#R

Deployment tooling
Summary: currently app servers sync l10nupdate from tin instead of using the rsync proxies just like scap does. BitTorrent?!!
 * [Ops] LocalisationUpdate == useless Tin

Jenkins

 * (Antoine) All jobs depending on mediawiki core now have mediawiki/vendor cloned as well. Unblocks Bryan Davis changes to core logging
 * (Antoine, Timo) Nasty regression in Jenkins Git plugin under Trusty (fixed)
 * (Zeljkof, Tobi) Wmde browsertests jobs migrated \O/
 * (Antoine, Dan, Zeljkof) JJB macro to easily run bundle commands
 * (Timo) Labs slaves monitoring: https://integration.wikimedia.org/monitoring/
 * (Antoine) Integration job to assert our PHPUnit fork works with mw/core release branches / master
 * (Antoine, devs) pywikibot running python3.4 tests on Ubuntu Trusty
 * Proposal to merge JJB and Zuul config repositories (see QA list)
 * ACTION self deploy on +2

Beta cluster

 * Andrew has been too busy (migrating LDAP) to order new hardware for nightly cluster Still hoping to do that soon.
 * Nightly cluster need resources from the whole engineering team. (Antoine, Greg during 1/1)

Browser tests

 * (Dan) Checking in with MM team about MMV metrics and need (or not) to setup traffic shaping
 * http://multimedia-metrics.wmflabs.org/dashboards/mmv#media_viewer_vs_file_page-graphs-tab
 * (Dan) Helping Zero with first browser tests for the Zero Portal
 * (Dan) Moving ahead on environment abstraction layer
 * (Tobi, Zeljkof, Antoine) Jenkins Performance Plugin enabled on all jobs (see QA list)
 * (Antoine, Dan, Zeljkof) Firefox local browser tests have a race condition killing Xvfb

Vagrant

 * (Dan) Draft of survey is in Qualtrics (needs final feedback)
 * (Dan) Tech Talk in November with Bryan Davis
 * Install party at Dev Summit: https://www.mediawiki.org/wiki/MediaWiki_Developer_Summit_2015#Workshops

Hiring

 * Elena starts on Tuesday (SF based)

Vacations/Confs/etc

 * 10/8: Zeljko - Croation Holiday
 * 10/6 - 10/10 Chris vacation
 * 10/13 - 10/17 Greg at MediaWiki Core offsite
 * 10/20 - 10/30 - Antoine might skip morning and work during evening {european pov}.
 * 10/21-22-23 Chris in SF (Elisabeth Hendrickson talk at WMF Oct 22)
 * 10/27 - 10/29 Chris at Google Test Automation Conference Seattle
 * 10/30 - 10/31 Chris vacation
 * 11/3 - 11/7: Antoine - OpenStack Summit Paris
 * 11/11 - Antoine Holiday (WW1)

= 2014-09-30 =

Team Business

 * Metrics
 * Should we set up a labs instance to capture and store metrics? (crons + some db + limn)
 * Registration for MediaWiki Developers Summit
 * Sam, Antoine, Mukunda :)
 * Ideas for Saturday during All Hands?
 * http://www.computerhistory.org/
 * http://www.bayareabrewerytours.com/

Scrum of Scrums

 * Dependency wall:

Phabricator

 * Quim Discovered a couple of issues with our 'secure' task hiding
 * One is a weekness that could expose private tasks via herald: https://phabricator.wikimedia.org/T493
 * This is getting addressed upstream thanks to chase's proposal to epriestley: https://secure.phabricator.com/T6211
 * Another issue was that the reporter of an issue wasn't actually able to access the maniphest task once it got submitted. https://phabricator.wikimedia.org/T475
 * This is fixed by https://gerrit.wikimedia.org/r/#/c/163753/
 * Still working on https://phabricator.wikimedia.org/T419 and https://phabricator.wikimedia.org/T169

Deployment tooling

 * Elasticsearch upgraded on logstash100[1-3] to match production. Other packages upgraded etc. Hopefully increase stability. logstash upgrades to come per beta below (reedy)

Jenkins

 * ✅ Zuul cloner bug, https://bugzilla.wikimedia.org/show_bug.cgi?id=71133
 * should the bug be FIX/RESO? :)
 * MediaWiki extensions qunit jobs migrated to it but VisualEditor. New job pending testing by VE team. (Antoine, Timo)
 * Wrote a diagnostic tool for Zuul (zuul-gearman.py). Doc updated at https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Debugging need moaar doc
 * Zuul stop processing jobs from time to time. Gathered traces which indicates it is most probably an issue in Gearman server.
 * C Scott proposed to merge zuul-config and jenkins-job-builder-config repositories. Thoughts?
 * Zeljkof, +1
 * Antoine to fill a bug about it and handle the merging + updating related jobs.

Beta cluster

 * logstash upgraded to 1.4.2 for testing, prior to deployment to production (reedy)
 * addition of redis yesterday broke beta labs https://bugzilla.wikimedia.org/show_bug.cgi?id=71415 bug was closed but re-opened because editing is still busted and Preferences also (at least)
 * greg to email deployers...

Browser tests

 * major updates to Echo and Flow repos in process
 * basic training for Rummana today
 * i10n screenshots (with Amir and Vikas)
 * some tests failing but he's coming back/still around
 * the missing font issue (solved)
 * user selectable language (default is a long list)
 * JJB browser tests cucumber macro needs evil refactoring. Been copy pasted all over the place.

Vagrant

 * Survey is going out this week? hopefully, once Dan's better and can get it in Qualtrics

Hiring

 * Elena Tonkovidova starts 14 October. Bay Area based.

Other
Selenium Workshop
 * Zeljkof working with Nik
 * asked Rachel for room/etc

Vacations/Confs/etc

 * 10/3: Zeljko - Conference
 * 10/8: Zeljko - Croation Holiday
 * 10/6 - 10/10 Chris vacation
 * 10/20 - 10/30 - Antoine might skip morning and work during evening {european pov}.
 * 10/21-22-23 Chris in SF (Elisabeth Hendrickson talk at WMF Oct 22)
 * 10/27 - 10/29 Chris at Google Test Automation Conference Seattle
 * 10/30 - 10/31 Chris vacation
 * 11/3 - 11/7: Antoine - OpenStack Summit Paris
 * 11/11 - Antoine Holiday (WW1)

= 2014-09-23 =

Team Business
From mail Antoine sent on RelEng list (corrected): Mon 19 - FREE (US holiday) / Travel day Tue 20 - FREE Wed 21 - All Hands Thu 22 - All Hands Fri 23 - Usual work day at office Sat-Sun: Free || offsite || team socializing at Alcatraz Mon 26th - tech days Tue 27th - tech days Wed 28th - FREE Thu 29th - FREE Fri 30th - FREE Follow up on mailing list
 * FOSS OPW projects? (similar to Google Summer of Code but restricted to ladies)
 * modernize rspec?
 * TODO: Chris to write up the idea (with Zeljko)
 * All Hands/Offsite current thinking:
 * 1 day at All Hands (Tuesday, the day before)
 * Week before Paris Hackathon
 * antoine: Wanna come to Nantes? :-]  maybe ;)   Also, why are you the same color?!
 * What's happening in Paris? yearly european hackathon like Zurich last year. Ohhh. i was thinking something "sooner" :D

Scrum of Scrums
The proxy itself is like OWASP Zed or whatever, we want to create the ability to send Selenium traffic to that particular proxy.
 * Dependency wall: (stupid mingle url...)
 * Chris/Dan card #135 https://wikimedia.mingle.thoughtworks.com/projects/scrum_of_scrums/cards/135

Phabricator

 * stuffs

Deployment tooling

 * scap stuffs

Jenkins

 * MediaWiki jobs switched to Zuul cloner. Now use the proper branch (was always using 'master')
 * Random fails of extensions patches against wmf branches.
 * Bug in Zuul cloner https://bugzilla.wikimedia.org/show_bug.cgi?id=71133 patch pending test + deploy

Beta cluster

 * monitoring based on graphite / Shinken. See YuviPanda announcements
 * http://shinken.wmflabs.org/host/beta-cluster (guest/guest)
 * http://graphite.wmflabs.org/ ( Look in the left tree for Graphite -> deployment-prep, all instances have metrics generated by Diamond)
 * Notifications:
 * IRC to #wikimedia-qa already
 * emails sent to a few people ACTION: need more people to be notified and act
 * TODO: Greg to sync up with the potential deployers

Browser tests

 * Pretty much everything that should be passing is passing
 * Throttling executors on Jenkins has improved pass rate
 * Chris is combing through Flow/Echo repos (and a little bit of MobileFrontend) doing refactoring and education per the quarterly goal
 * Chris would like to set up pairing sessions in SF Oct 22/23
 * MMV tests are driving performance measurement
 * Getting them up and running on a new labs instance (multimedia-perf.eqiad.wmflabs)
 * SauceLabs is likely culprit of inaccurate metrics
 * Yslow related, bug asking to add it as a job to run on patchset proposal https://bugzilla.wikimedia.org/show_bug.cgi?id=57137

Vagrant

 * Spun up a new labs instance for MMV performance testing using MWV (yay)
 * Finalize and send out the survey!
 * Researching:
 * Lightweight monitoring
 * resource monitoring
 * auto bug report helper
 * anonymous reporting
 * cookies & donuts?

Hiring


Vacations/Confs/etc

 * 10/3: Zeljko - Conference
 * 10/8: Zeljko - Croation Holiday
 * 10/6 - 10/10 Chris vacation
 * 10/20 - 10/30 - Antoine might skip morning and work during evening {european pov}.
 * 10/27 - 10/29 Chris at Google Test Automation Conference Seattle
 * 10/30 - 10/31 Chris vacation
 * 11/3 - 11/7: Antoine - OpenStack Summit Paris
 * 11/11 - Antoine Holiday (WW1)

= 2014-09-16 =

Phabricator

 * redirection scripts (BZ urls to Phab urls)
 * almost finished
 * Just waiting on launch, will work with chase to deploy it
 * the gerrit -> phab bot
 * Java :)
 * ask for help from Nik/Chad/Christian

Deployment tooling

 * l10n and scap colliding
 * https://bugzilla.wikimedia.org/show_bug.cgi?id=70446

Jenkins

 * extensions now being tested with:
 * proper mediawiki/core branch (deployed today)
 * mediawiki/vendor
 * Still have to migrate the extensions -qunit jobs (WIP)
 * Wikidata related jobs partly reintegrated on Wikimedia Jenkins
 * Next items:
 * early adopt phabricator
 * isolating tests using labs infra.
 * merge zuul-config and jjb-config repositories (suggested by cscott)
 * auto deploy CI related changes on +2
 * Anyone interested in some Jenkins training ? ( Timezones sucks :-/ )
 * Chris started re-doing docs on mw.o. Starting with getting rid of references to Cloudbees.
 * Jenkins perf improvements
 * plan a sane master/slave arrangements
 * Design load structure
 * TODO: GREG to find find out Timo's involvement

Beta cluster
Antoine: both are huge additions with a long record of proven success.
 * jeremyb rampage
 * matanya granted root on beta (ops & puppet volunteer)
 * Search slowness; https://bugzilla.wikimedia.org/show_bug.cgi?id=70869
 * We have monitoring on beta cluster thanks to Yuvi!
 * Wait on second cluster

Browser tests

 * early WIP of environment abstration layer
 * https://gerrit.wikimedia.org/r/#/c/159644/
 * Geolocation use case? Chris to send email I think :)
 * helping MMV with performance testing using MW-Selenium and an "isolated" labs instance
 * Chris to refactor Echo tests first in conjunction with corefeatures team

Vagrant

 * helping MMV with performance testing using MW-Selenium and an "isolated" labs instance
 * looking for ways to setup a "traffic shaper" role in MWV (using `tc` perhaps)
 * not sure how to achieve more isolation in labs (bigger instance == more dedicated?)
 * need to finalize MWV survey
 * https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Vagrant_survey

Hiring
HR.....................................................