Quality Assurance/Strategy

Strategy for QA for 2012-13 May 2, 2012 ChrisM, Sumana, RobLa, ErikM


 * Volunteer strategy (outreach & coordination):
 * outside community first - sophisticated testing of shallow components that experienced community would miss
 * Weekend Testers
 * step in the direction of attracting expert testers, get them into the WM community
 * a series of high-profile, collaborative testing exercises, like Mozilla's test days, after we've collaborated with others
 * then, engage with Wikimedia community for deep testing of i18n, ProofreadPage, & other components that outsiders wouldn't hit
 * CentralNotice?
 * What else?
 * Have a central "these are our QA priorities" page (test plan for Saturday is a good start)
 * announce WMF institutional test days (after the outside community gets engaged)
 * the day after deployment to non-Wikipedias with the new version
 * the day after small deployments (mobile, AFT, other incremental updates)
 * And we need to communicate better about these deployments! (we need a more sustainable release notes process.... difficult to know what's significant enough to include and what's not.  To discuss re commit messages - Erik putting some thoughts together.)
 * consider test case management, like Litmus & Moztrap/Mousetrap?
 * considering just using freeform wiki pages. Humans for exploratory testing in combination with automation for more scriptable actions, crossbrowser stuff.
 * Follow-up after events?
 * We will set up an opt-in mailing list before this weekend, and Chris & volunteer QA coordinator will publicize future test events/needs to that list
 * For this event, Chris will record name & some kind of unique identifier for each person who has an interaction with him - that way he can track how many people just keep showing up, of their own accord, and then we can run the experiment of more active, personal followup after a future event


 * Test automation strategy
 * We have a huge dev-to-QA ratio; work will have to be mostly by devs
 * Selenium more Chris's area... write tests that are robust, will last for a few years
 * How do we make it easy to write a unit test?
 * Do the tests run against a production or a test environment? Example: you can test ProofreadPage, 99%, without being logged in, and it's always there.  And Labs is not robust enough.  Doesn't return pages fast enough.  Labs needs a reliable automated deploy process.
 * Think about UploadWizard, which is fragile and breaks every release! Need a Selenium test for that, or is this too deep for automation?  There would be at least 1 basic test.  Core piece of Commons, which has strategic importance.  But how far do we go?  Every media type?  No. Just get a basic test: get logged in, make 2 or 3 styles of edits, upload an image, 1 or 2 for Wikisource, 1 or 2 for Wikinews, a FlaggedRevs test
 * Note that any test involving logging in will have to involve a user that can't do damage, since the user/password will probably be public. Also note that any automated browser test that uses files on disk (like for uploading) will be subject to a whole lot of extra maintenance, since the automated test should run in/from any suitable environment: labs, production, local.
 * If you're concerned about a path through an app, that's a good candidate for a browser test. UploadWizard is a great candidate.  But the tough thing is managing files on disk.
 * A zillion little elements that each do a diff. thing in the UI is tough. Example: ProofreadPage. Hard to test with a unit test, makes sense with a browser test.
 * Focusing - what tests are easier to write and maintain over time? PP - we can really use a minimal level of smoke testing.  Focus on what a lot of people see but we don't get reports on.  Like, major breakage in IE7.  People just leave and do not come back.
 * What do we use? We can't use the VMs in the office. Something like Sauce is probably ultimately our best bet.  Biggest concern: getting useful tests in place in the first place.  Then figure out where to host them.
 * Ruby still makes a lot of sense.
 * Selenium tests in Ruby? Selenium WebDriver is in the process of being approved by W3C as a standard - IE, Firefox, Chrome, & Opera have signed on. Languages: Python, C#, Java, & Ruby.  The Python stuff (see Mozilla for example) requires a lot of custom scaffolding we'd have to write ourselves.
 * Outside unit tests & Selenium tests, are we looking into any other kind of testing? Benchmark or smoke tests? We don't have reliable ongoing data on performance on, e.g., uploads.

Follow up: Jenkins