User talk:Cmcmahon

Talking

Cmcmahon 20:29, 7 February 2012 (UTC)

Community Testing/QA
It would benefit WMF to have interested parties test various aspects of Mediawiki, Wikipedia, and related projects in the user interface, with various browsers, before they are released to the general public.

As I see it, the issue to be addressed is not to persuade more core/Extensions/Gadgets people to do more testing. Nor is the issue to attract more FLOSS community members to Mediawiki/Wikipedia testing.

Instead, the issue is to attract more people from the general software testing community to the effort to test Mediawiki/Wikipedia, particularly at a per-feature and per-release level. And, while doing so, to keep the amount of noise (NOTANISSUE, WONTFIX, etc.) to a minimum.

Two reasonable approaches for doing this suggest themselves.

Feature Testing by Exploratory Testers
For a number of features, it is desirable to have as many people as possible investigate the feature, looking for issues and problems in various browser/OS/data/etc. combinations before such features are deployed to production.

An excellent example of such a potential testing project would be the Timed Media Handler (TMH). TMH has many parameters, ranging from the display of media files in various browsers, to the interaction of media files on the wiki(s) itself, to the nature of the files being uploaded.

However, it is not feasible that a community tester (especially Windows users) would be in a position to set up a local test environment for TMH. A shared test environment for features such as TMH is highly desirable.

Here is an excellent example from Mozilla: https://quality.mozilla.org/2012/03/web-qa-testday-for-the-affiliates-project-march-29-2012/ They have in place a dedicated test environment, an etherpad with test charters, and Bugzilla all set up to handle bug reports. Mozilla has "Testdays" of several different kinds: https://quality.mozilla.org/category/events/month

Automated Browser/UI/Mobile Testing
WMF has no automated browser testing in place right now. This is actually an advantage, because the general environment for doing such testing has seen a surge of innovation in recent times, and those involved are forming consensus on the best approaches to doing so.

Selenium is far and away the tool of choice for doing browser automation. The Selenium bindings are maintained institutionally for four languages: Java, C#, Python, and Ruby. Java and C# are poor choices for WMF.

Between Python and Ruby, I would strongly recommend Ruby, for a number of reasons.


 * The culture of testing is strong in Ruby, less so for Python.
 * The Ruby/Selenium community is in strong agreement as to optimal approaches to browser automation: Page Objects, RSpec, rake, etc. the "gems" involved are all well-known and well-understood by the Ruby testing community.
 * There is a strong culture of philanthropy in the Ruby community. I've already had an offer to do a 2-day "GiveCamp" to get the WMF browser automation project underway, or to improve it once it exists.
 * Mozilla has the attention of the Python browser testing community. No project as famous as Wikipedia has tapped the Ruby testing community, which is potentially much larger and more expert than the equivalent Python community.
 * The way Page Objects are implemented in Ruby also supports Mobile testing. http://watirwebdriver.com/mobile-devices/ (Note: Watir today is a higher-level API that wraps Selenium 2.0)

Job #1: Reduce Barriers to Entry
Whether for investigation of features before release, or for an ongoing browser automation effort, WMF should make a strong effort to reduce the barriers to entry for anyone in the community that wants to contribute to the testing effort.


 * Good shared test environment(s)
 * make Labs useful to strangers
 * Have the correct version of MW in place.
 * Have the test wiki(s) configured properly (e.g. currently there is no working link to upload a file on beta Commons, one has to know the URL for UploadWizard.)
 * Create test envs other than labs beta?
 * encapsulate Mediawiki for download e.g. Bitnami? (nice to have)


 * Prepare
 * Relevant documentation (test plan, etc.) in place before start of testing
 * Modern, attractive frameworks to automate with
 * Support for testers via IRC, mail, etc. (i.e. Mozilla)
 * Have Bugzilla set up for Community QA reporting, with proper categories and a published procedure for reporting issues found