User talk:Cmcmahon

Cmcmahon 20:29, 7 February 2012 (UTC)

Community Testing/QA
It would benefit WMF to have interested parties test various aspects of Mediawiki, Wikipedia, and related projects in the user interface, with various browsers, before they are released to the general public.

As I see it, the issue to be addressed is not to persuade more core/Extensions/Gadgets people to do more testing. Nor is the issue to attract more FLOSS community members to Mediawiki/Wikipedia testing.

Instead, the issue is to attract more people from the general software testing community to the effort to test Mediawiki/Wikipedia, particularly at a per-feature and per-release level. And, while doing so, to keep the amount of noise (NOTANISSUE, WONTFIX, etc.) to a minimum.

Two reasonable approaches for doing this suggest themselves.

Feature Testing by Exploratory Testers
For a number of features, it is desirable to have as many people as possible investigate the feature, looking for issues and problems in various browser/OS/data/etc. combinations before such features are deployed to production.

An excellent example of such a potential testing project would be the Timed Media Handler (TMH). TMH has many parameters, ranging from the display of media files in various browsers, to the interaction of media files on the wiki(s) itself, to the nature of the files being uploaded.

However, it is not feasible that a community tester (especially Windows users) would be in a position to set up a local test environment for TMH. A shared test environment for features such as TMH is highly desirable.

Here is an excellent example from Mozilla: https://quality.mozilla.org/2012/03/web-qa-testday-for-the-affiliates-project-march-29-2012/ They have in place a dedicated test environment, an etherpad with test charters, and Bugzilla all set up to handle bug reports. Mozilla has "Testdays" of several different kinds: https://quality.mozilla.org/category/events/month

Note: Weekend Testing http://weekendtesting.com/ is a group of about 5000 Exploratory Testers in chapters in India, Americas, Europe, and Australia/NZ who gather on the first Saturday of every month (Americas at least) to do ET on some sort of publicly available software. They would be an excellent resource both to do feature/release testing and also to publicize the WMF community QA effort.

Automated Browser/UI/Mobile Testing
WMF has no automated browser testing in place right now. This is actually an advantage, because the general environment for doing such testing has seen a surge of innovation in recent times, and those involved are forming consensus on the best approaches to doing so.

Selenium is far and away the tool of choice for doing browser automation. The Selenium bindings are maintained institutionally for four languages: Java, C#, Python, and Ruby. Java and C# are poor choices for WMF.

Between Python and Ruby, I would strongly recommend Ruby, for a number of reasons.


 * The culture of testing is strong in Ruby, less so for Python.
 * The Ruby/Selenium community is in strong agreement as to optimal approaches to browser automation: Page Objects, RSpec, rake, etc. the "gems" involved are all well-known and well-understood by the Ruby testing community.
 * There is a strong culture of philanthropy in the Ruby community. I've already had an offer to do a 2-day "GiveCamp" to get the WMF browser automation project underway, or to improve it once it exists.
 * Mozilla has the attention of the Python browser testing community. No project as famous as Wikipedia has tapped the Ruby testing community, which is potentially much larger and more expert than the equivalent Python community.
 * The way Page Objects are implemented in Ruby also supports Mobile testing. http://watirwebdriver.com/mobile-devices/ (Note: Watir today is a higher-level API that wraps Selenium 2.0)

Job #1: Reduce Barriers to Entry
Whether for investigation of features before release, or for an ongoing browser automation effort, WMF should make a strong effort to reduce the barriers to entry for anyone in the community that wants to contribute to the testing effort.


 * Good shared test environment(s)
 * make Labs useful to strangers
 * Have the correct version of MW in place.
 * Have the test wiki(s) configured properly (e.g. currently there is no working link to upload a file on beta Commons, one has to know the URL for UploadWizard.)
 * Create test envs other than labs beta?
 * encapsulate Mediawiki for download e.g. Bitnami? (nice to have)


 * Prepare
 * Relevant documentation (test plan, etc.) in place before start of testing
 * Modern, attractive frameworks to automate with
 * Support for testers via IRC, mail, etc. (i.e. Mozilla)
 * Have Bugzilla set up for Community QA reporting, with proper categories and a published procedure for reporting issues found

Selenium/Ruby project
Prequisites for Awesome Selenium Testing

Framework


 * Ruby
 * selenium-webdriver
 * watir-webdriver
 * page-object
 * rspec
 * rake

See https://github.com/chrismcmahon/Page-Object-WMF-spike for reference implementation.

WMF host


 * RVM/gems, see http://ryanbigg.com/2010/12/ubuntu-ruby-rvm-rails-and-you/

After doing some research on the apt vs. gems issue, it seems like the approach most of the Ruby community takes is to keep apt and Ruby completely separate. In practice this means getting RVM (Ruby Version Manager) from github, installing it, and then using RVM to install and manage Ruby versions and gems. The link above describes the practice in general. I tested this out on a labs instance and it worked perfectly. RVM also solves the issue of maintaining Ruby infrastructure without sudo; and since RVM contains everything within user space, not root, it's more secure.

I polled some Rubyists in the testing community: Convio has RVM/gems on their prod servers. Groupon has RVM/gems in their CI and staging envs, and for prod they have abandoned apt altogether and use their own package manager.

Test Environment


 * http://en.wikipedia.beta.wmflabs.org/
 * http://commons.wikipedia.beta.wmflabs.org/
 * Windows VMs with JRE/selenium-server
 * Other OS VMs? (Ubuntu)

Test Data Management


 * Need strategy for test user with publicly available user/pass who can log in on labs beta wikis
 * Possibly need scheme to reset test data, dbs, etc.

Jenkins stuff


 * Deploy from trunk or release candidate to labs beta wikis
 * Kick off rake task, gather results

Everything happens in the browser

 * Don't manipulate db tables
 * Don't manipulate files on disk

No hacks

 * Every aspect of the test framework should Just Work

Let the user write tests out of the box

 * get the framework/scaffolding out of the way, out of sight

Defined test environments

 * Test suite for clean install
 * Test suite for maintained shared test env. e.g. Beta Labs

Dynamic test data

 * Get test data at runtime
 * Extensions can help
 * If necessary create REST API exposing XML for browser to read
 * Note: data must be represented as an 'attribute' of the XML node(s) because IE is retarded


 * http://www.mediawiki.org/wiki/Nuke could be helpful

Issues

 * Static test data
 * What to do about user/pass for tests when logged in?
 * (Looks like http://www.mediawiki.org/wiki/Extension:Sudo might do the job)
 * (Alternately, make the test user password public but only allow it highly limited actions: https://www.mediawiki.org/wiki/Manual:User_rights)

Weekend Testing Americas May 5 Test Plan
WTA May 5 Test Plan


 * Objectives


 * Test MediaWiki version 1.20wmf02 against version 1.20wmf01
 * Regression testing for 1.20wmf01
 * General Exploratory Testing in various browsers


 * Platforms/Browsers of Interest


 * IE7 and IE8/IE9 in "compatibility mode"
 * IE8
 * IE9
 * Firefox with and without "HTTPS-Everywhere" extension
 * Chrome
 * All browsers with and without "privacy/incognito" mode set


 * Test Environments (Note: we are testing in production environments, please be respectful, see Recommended Procedures below)


 * 1.20wmf01: All Wikipedia sites, e.g. http://en.wikipedia.org/
 * 1.20wmf02: All non-Wikipedia sites (Wiktionary, Wikisource, Wikinews, Wikibooks, Wikiquote, Wikiversity, and other sites)
 * http://en.wikisource.org/ is of interest because ProofreadPage has many moving parts
 * http://commons.wikimedia.org/ is also of interest.


 * Setup


 * Create a global account on any wiki and login. This is not required, but it makes several things easier:
 * Any edits to pages preserves the identity of the editor. This is of particular interest for the Sandbox page
 * Provides a User_talk page, which is a good place to do testing
 * Allows persistent setting of various user preferences relevant to testing


 * Create an account in Bugzilla https://bugzilla.wikimedia.org/
 * This is also not strictly required, but recommended strongly for reporting issues


 * Test Charters and Features to be Tested


 * 1.20wmf01 test charters: http://www.mediawiki.org/wiki/Mediawiki_1.20_Feature_Test_Charters
 * 1.20wmf01 Features: http://www.mediawiki.org/wiki/MediaWiki_1.20/wmf1/overview
 * 1.20wmf01 Release Notes: https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=blob;f=RELEASE-NOTES-1.20;h=0e8cdc09e6163154a6182e301a37aebdb244cc41;hb=refs/heads/master


 * 1.20wmf02 test charters: http://www.mediawiki.org/wiki/Mediawiki_1.20wmf2_Feature_Test_Charters


 * Recommended Procedures


 * Choose a test charter or feature from the 1.20wmf02 list
 * Perform ET for that charter or feature on a non-Wikipedia site
 * Do ET for similar actions on a Wikipedia site with 1.20wmf01 installed


 * Choose a test charter from 1.20wmf01 list
 * Investigate both Wikipedia and non-Wikipedia sites for proper function of that feature


 * Note: if you are unsure of what version of MediaWiki is installed in your environment, most wikis have the page Special:Version in place, for example http://en.wikipedia.org/wiki/Special:Version


 * Note: instructions to use Sandbox pages are here: http://en.wikipedia.org/wiki/Wikipedia:Sandbox . An example of a User talk page is here: http://en.wikipedia.org/wiki/User_talk:Cmcmahon


 * Reporting issues


 * When creating a bugzilla ticket, make the title "WTA " for help in triage later
 * Discuss issues in WTA Skype channel and/or IRC channel freenode#wikimedia-dev
 * optional: Edit http://www.mediawiki.org/wiki/User_talk:Cmcmahon with description of the issue
 * optional: Email cmcmahon at wikimedia dot org


 * TODO
 * replace TBD

1.20wmf2 research
ahead of official RELEASE NOTES:

in wmf2 not in wmf1
 * Added GitViewers hook for extensions using external git repositories to have a web-based repository viewer linked to from Special:Version.
 * Memcached debug logs can now be sent to their own file logs by setting $wgDebugLogFile['memcached'] to some filepath.
 * (bug 35685) api.php URL and other entry point URLs are now listed on Special:Version
 * Edit notices can now be translated.
 * (bug 22887) Add warning and tracking category for preprocessor errors
 * (bug 34428) Fixed incorrect hash mismatch errors in the DiffHistoryBlob
 * (bug 34929) Show the correct diff when a section edit is rejected by the spam
 * jQuery UI upgraded to 1.8.19.
 * (bug 35572) Blocks appear to succeed even if query fails due to wrong DB structure
 * (bug 31757) Add a word-separator between help-messages in HTMLForm
 * (bug 30410) Removed deprecated $wgFilterCallback and the 'filtered' API error.
 * (bug 36012) Space in $separatorTransformTable should be non-breaking in Portuguese, Esperanto and Udmurt.

1.20wmf2 Feature Test Charters: DRAFT IN PROGRESS


 * Investigate diff displays
 * Trigger spam on edits; check diffs (spam: ALL CAPS, drug names, obscenity, etc. USE SANDBOX OR USER TALK PAGES! )
 * Validate Special:Version display and data
 * Validate appearance and content of Help messages in various locations
 * Investigate edit notices in languages other than English
 * Portuguese, Esperanto and Udmurt of particular interest

OpenHatch Article Feedback Tool 9 June Test Plan

 * Test Charters and Features to be Tested


 * Test nearly-final version of Article Feedback Tool. Last chance to identify potential issues before final version


 * Validate appearance and behavior of AFT user interface to create feedback on articles.
 * Validate appearance and display of existing feedback for articles
 * Validate workflow that leads to editing WikiPedia
 * Validate filtering of feedback


 * Platforms/Browsers of Interest


 * IE7 and IE8/IE9 in "compatibility mode"
 * IE8
 * IE9
 * IE10
 * Firefox with and without "HTTPS-Everywhere" extension
 * Chrome
 * All browsers with and without "privacy/incognito" mode set


 * Test Environments (Note: we are testing in production environments, please be respectful)


 * For experiments with inappropriate feedback use the Golden-crowned Sparrow page: http://en.wikipedia.org/wiki/Golden-crowned_Sparrow?aftv5_form=1&aftv5_link=E and http://en.wikipedia.org/wiki/Special:ArticleFeedbackv5/Golden-crowned_Sparrow?aftv5_form=1 (This is something of a "throwaway" page, used for testing by the AFT team)


 * For experiments with high volume and/or controversy use the Barack Obama page: http://en.wikipedia.org/wiki/Barack_Obama?aftv5_form=1&aftv5_link=E and http://en.wikipedia.org/wiki/Special:ArticleFeedbackv5/Barack_Obama


 * For a non-controversial article see http://en.wikipedia.org/wiki/Drew_Barrymore?aftv5_form=1&aftv5_link=E and http://en.wikipedia.org/wiki/Special:ArticleFeedbackv5/Drew_Barrymore


 * Setup


 * IMPORTANT: If possible, create an account on http://en.wikipedia.org/ no later than Tuesday 5 June and complete at least ten edits.  This makes your account "autoconfirmed" and provides access to more AFT features.  See http://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#Access_and_permissions for details


 * Create a global account on http://en.wikipedia.org/ and login. This is not required, but it makes several things easier:
 * Any edits to pages preserves the identity of the editor.
 * Provides a User_talk page, which is a good place to take notes
 * Allows persistent setting of various user preferences relevant to testing


 * Create an account in Bugzilla https://bugzilla.wikimedia.org/
 * This is also not strictly required, but recommended strongly for reporting issues


 * Procedures (need screenshots)


 * For tracking purposes, use "OHAFT " when appropriate when creating data.


 * Create feedback
 * Use unusual characters, capitalization, punctuation, etc.


 * Display feedback
 * Check links on page
 * Check 'Show more..'
 * Check display of long text, unusual characters, etc.


 * Workflow
 * Check paths through AFT features, back button, etc.
 * Check ultimate ability to edit WikiPedia


 * Filtering (admin task)
 * Feedback containing unusual characters, links, etc. to be analyzed by admins and AFT team


 * Reporting issues


 * When creating a bugzilla ticket, make the title "OHAFT " for help in triage later (Bugzilla: https://bugzilla.wikimedia.org/)
 * Discuss issues in IRC channel freenode#mediawiki
 * optional: Edit http://www.mediawiki.org/wiki/User_talk:Cmcmahon with description of the issue
 * optional: Email cmcmahon at wikimedia dot org


 * Technical Resources
 * http://www.mediawiki.org/wiki/Article_feedback/Version_5