Quality Assurance/How we prioritize

QA meeting Feb 26 -- how do we prioritize what we work on, re testing?

Attending:
 * originally: QA was Chris:
 * Michelle Grover for mobile
 * Zeljko for test automation
 * Quim for community events

Chris: overview of QA activities

Projects they've helped with: (13 entries) Places:
 * AFTv5 - E2
 * Echo - E2
 * Page Triage - E2
 * VE
 * Mobile
 * Mobile
 * Language
 * Guided Tour - E3
 * Account Creation - E3
 * Math
 * UploadWizard
 * TMH
 * Search
 * PDBHandler - Community
 * Environments: (11 entries)
 * labs host (ee prototype)
 * beta cluster (AFT, mobile)
 * caching (Antoine says, "like 99% of our infrastructure is about caching so caching is the top root cause of issues :-]")
 * search <--- Lucene could be setup in beta
 * jobqueue
 * thumbnails
 * Lucene
 * bits.wikimedia.org
 * parsercache
 * ResourceLoader
 * swift
 * mediawiki.org (echo)
 * enwiki/prod (ve, e3)
 * Test2wiki (upload wizard, math, etc.)
 * future: vagrant

Exploratory:
 * Staff
 * Pro volunteers
 * Wiki volunteers

Automated
 * backlog!
 * needs subject matter experts (to write in plain english the testing needs)

QA Weekly goals: E2 wants to be able to test stuff before it is merged -- beta cluster for non-master branches is being worked on.
 * http://www.mediawiki.org/wiki/QA/Weekly_goals
 * https://www.mediawiki.org/wiki/QA/Review_February_2013#Notes_from_QA_Quarterly_review_-_2013-02-08
 * See "How do we decide what to prioritize? Stuff where it's useful to test before production." discussion
 * also see "How do we decide what to focus exploratory testing energy on? How do we balance it on a per-area basis? How do we get those teams to accept the  help?" section
 * Outcome: things that would benefit from more software testing?
 * needs to know what needs to be tested before projection?

matrix: Y - features, X- types of testing

X-axis - Types of testing: Prioritization matrix: https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0Agte_lJNpi-OdGpJSHdMZV83UjdTdHc5UFVrdXBwaWc#gid=0 Want from features: Current idea of process: build out this spreadsheet, see that it has all our engineering projects & what kinds of testing they're amenable to, and regularly check in to update it with Howie/Chris/Erik.
 * test environment creation
 * Automatic
 * unit testing (jenkins & testswarm) -- skip for today
 * integration testing (Jenkins & testswarm) -- skip for today
 * automated browser testing (can stop because code is bad, or test environement broken, deployment broken). More for stable stuff?
 * configuration testing (make sure it works on all browsers and operating systems)
 * Exploratory testing
 * developer/product manager exploratory testing - manual browser testing, acceptance testing
 * volunteer exploratory testing
 * QA staff exploratory testing (ex. AFT for Chris, Michelle for mobile)
 * 1 exploratory testing project/month
 * 1 automated testing project/month
 * synchronize focuses with community/communications activities (blog posts, community events, etc.)