User talk:Cmcmahon

Cmcmahon 20:29, 7 February 2012 (UTC)

Community Testing/QA
It would benefit WMF to have interested parties test various aspects of Mediawiki, Wikipedia, and related projects in the user interface, with various browsers, before they are released to the general public.

As I see it, the issue to be addressed is not to persuade more core/Extensions/Gadgets people to do more testing. Nor is the issue to attract more FLOSS community members to Mediawiki/Wikipedia testing.

Instead, the issue is to attract more people from the general software testing community to the effort to test Mediawiki/Wikipedia, particularly at a per-feature and per-release level. And, while doing so, to keep the amount of noise (NOTANISSUE, WONTFIX, etc.) to a minimum.

Two reasonable approaches for doing this suggest themselves.

Feature Testing by Exploratory Testers
For a number of features, it is desirable to have as many people as possible investigate the feature, looking for issues and problems in various browser/OS/data/etc. combinations before such features are deployed to production.

An excellent example of such a potential testing project would be the Timed Media Handler (TMH). TMH has many parameters, ranging from the display of media files in various browsers, to the interaction of media files on the wiki(s) itself, to the nature of the files being uploaded.

However, it is not feasible that a community tester (especially Windows users) would be in a position to set up a local test environment for TMH. A shared test environment for features such as TMH is highly desirable.

Here is an excellent example from Mozilla: https://quality.mozilla.org/2012/03/web-qa-testday-for-the-affiliates-project-march-29-2012/ They have in place a dedicated test environment, an etherpad with test charters, and Bugzilla all set up to handle bug reports. Mozilla has "Testdays" of several different kinds: https://quality.mozilla.org/category/events/month

Note: Weekend Testing http://weekendtesting.com/ is a group of about 5000 Exploratory Testers in chapters in India, Americas, Europe, and Australia/NZ who gather on the first Saturday of every month (Americas at least) to do ET on some sort of publicly available software. They would be an excellent resource both to do feature/release testing and also to publicize the WMF community QA effort.

Automated Browser/UI/Mobile Testing
WMF has no automated browser testing in place right now. This is actually an advantage, because the general environment for doing such testing has seen a surge of innovation in recent times, and those involved are forming consensus on the best approaches to doing so.

Selenium is far and away the tool of choice for doing browser automation. The Selenium bindings are maintained institutionally for four languages: Java, C#, Python, and Ruby. Java and C# are poor choices for WMF.

Between Python and Ruby, I would strongly recommend Ruby, for a number of reasons.


 * The culture of testing is strong in Ruby, less so for Python.
 * The Ruby/Selenium community is in strong agreement as to optimal approaches to browser automation: Page Objects, RSpec, rake, etc. the "gems" involved are all well-known and well-understood by the Ruby testing community.
 * There is a strong culture of philanthropy in the Ruby community. I've already had an offer to do a 2-day "GiveCamp" to get the WMF browser automation project underway, or to improve it once it exists.
 * Mozilla has the attention of the Python browser testing community. No project as famous as Wikipedia has tapped the Ruby testing community, which is potentially much larger and more expert than the equivalent Python community.
 * The way Page Objects are implemented in Ruby also supports Mobile testing. http://watirwebdriver.com/mobile-devices/ (Note: Watir today is a higher-level API that wraps Selenium 2.0)

Job #1: Reduce Barriers to Entry
Whether for investigation of features before release, or for an ongoing browser automation effort, WMF should make a strong effort to reduce the barriers to entry for anyone in the community that wants to contribute to the testing effort.


 * Good shared test environment(s)
 * make Labs useful to strangers
 * Have the correct version of MW in place.
 * Have the test wiki(s) configured properly (e.g. currently there is no working link to upload a file on beta Commons, one has to know the URL for UploadWizard.)
 * Create test envs other than labs beta?
 * encapsulate Mediawiki for download e.g. Bitnami? (nice to have)


 * Prepare
 * Relevant documentation (test plan, etc.) in place before start of testing
 * Modern, attractive frameworks to automate with
 * Support for testers via IRC, mail, etc. (i.e. Mozilla)
 * Have Bugzilla set up for Community QA reporting, with proper categories and a published procedure for reporting issues found

Selenium/Ruby project
Prequisites for Awesome Selenium Testing

Framework


 * Ruby
 * selenium-webdriver
 * watir-webdriver
 * page-object
 * rspec
 * rake

See https://github.com/chrismcmahon/Page-Object-WMF-spike for reference implementation.

WMF host


 * RVM/gems, see http://ryanbigg.com/2010/12/ubuntu-ruby-rvm-rails-and-you/

After doing some research on the apt vs. gems issue, it seems like the approach most of the Ruby community takes is to keep apt and Ruby completely separate. In practice this means getting RVM (Ruby Version Manager) from github, installing it, and then using RVM to install and manage Ruby versions and gems. The link above describes the practice in general. I tested this out on a labs instance and it worked perfectly. RVM also solves the issue of maintaining Ruby infrastructure without sudo; and since RVM contains everything within user space, not root, it's more secure.

I polled some Rubyists in the testing community: Convio has RVM/gems on their prod servers. Groupon has RVM/gems in their CI and staging envs, and for prod they have abandoned apt altogether and use their own package manager.

Test Environment


 * http://en.wikipedia.beta.wmflabs.org/
 * http://commons.wikipedia.beta.wmflabs.org/
 * Windows VMs with JRE/selenium-server
 * Other OS VMs? (Ubuntu)

Test Data Management


 * Need strategy for test user with publicly available user/pass who can log in on labs beta wikis
 * Possibly need scheme to reset test data, dbs, etc.

Jenkins stuff


 * Deploy from trunk or release candidate to labs beta wikis
 * Kick off rake task, gather results

Everything happens in the browser

 * Don't manipulate db tables
 * Don't manipulate files on disk

No hacks

 * Every aspect of the test framework should Just Work

Let the user write tests out of the box

 * get the framework/scaffolding out of the way, out of sight

Defined test environments

 * Test suite for clean install
 * Test suite for maintained shared test env. e.g. Beta Labs

Dynamic test data

 * Get test data at runtime
 * Extensions can help
 * If necessary create REST API exposing XML for browser to read
 * Note: data must be represented as an 'attribute' of the XML node(s) because IE is retarded


 * http://www.mediawiki.org/wiki/Nuke could be helpful

Issues

 * Static test data
 * What to do about user/pass for tests when logged in?
 * (Looks like http://www.mediawiki.org/wiki/Extension:Sudo might do the job)
 * (Alternately, make the test user password public but only allow it highly limited actions: https://www.mediawiki.org/wiki/Manual:User_rights)

Weekend Testing Americas May 5 Test Plan
WTA May 5 Test Plan


 * Objective


 * Test MediaWiki version 1.20wmf02 against version 1.20wmf01
 * Regression testing for 1.20wmf01
 * General Exploratory Testing in various browsers


 * Platforms/Browsers of Interest


 * IE7 and IE8/IE9 in "compatibility mode"
 * IE8
 * IE9
 * Firefox with and without "HTTPS-Everywhere" extension
 * Chrome
 * All browsers with and without "privacy/incognito" mode set


 * Test Environments


 * 1.20wmf01: All Wikipedia sites, e.g. http://en.wikipedia.org/
 * 1.20wmf02: All non-Wikipedia sites (Wiktionary, Wikisource, Wikinews, Wikibooks, Wikiquote, Wikiversity, and other sites)


 * Setup


 * Create a global account on any wiki and login. This is not required, but it makes several things easier:
 * Any edits to pages preserves the identity of the editor. This is of particular interest for the Sandbox page
 * Provides a User_talk page, which is a good place to do testing
 * Allows persistent setting of various user preferences relevant to testing


 * Create an account in Bugzilla https://bugzilla.wikimedia.org/
 * This is also not strictly required, but recommended strongly for reporting issues


 * Test Charters and Features to be Tested


 * 1.20wmf01 test charters: http://www.mediawiki.org/wiki/Mediawiki_1.20_Feature_Test_Charters
 * 1.20wmf01 Features: http://www.mediawiki.org/wiki/MediaWiki_1.20/wmf1/overview
 * 1.20wmf01 Release Notes: https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=blob;f=RELEASE-NOTES-1.20;h=0e8cdc09e6163154a6182e301a37aebdb244cc41;hb=refs/heads/master


 * 1.20wmf02 TBD


 * Recommended Procedures


 * Choose a test charter or feature from the 1.20wmf02 list
 * Perform ET for that charter or feature on a non-Wikipedia site
 * Do ET for similar actions on a Wikipedia site with 1.20wmf01 installed


 * Choose a test charter from 1.20wmf01 list
 * Investigate both Wikipedia and non-Wikipedia sites for proper function of that feature


 * Reporting issues


 * When creating a bugzilla ticket, make the title "WTA " for help in triage later
 * Discuss issues in WTA Skype channel and/or IRC channel freenode#
 * Edit http://www.mediawiki.org/wiki/User_talk:Cmcmahon with description of the issue
 * Email cmcmahon at wikimedia dot org


 * TODO
 * replace TBD
 * call out role of Commons explicitly