Reading/Web/QA

We use manual testing as well as automated testing to QA our products.

Our manual testing frequently occurs as part of our sprint process. After code has gone through engineers and design we hand it over for QA.

Manual handover
First it is important to check we have instructions on how and where to test the change in the description, given we have many test environments identifying where testing can take place.

We have 2 environmentsː


 * The beta cluster, which updates frequently as code is merged
 * reading-web-staging.wmflabs.org, which is specialized for testing article content

Depending on what is being tested usually one is more suitable than the other. It's extremely important to clarify where to test to avoid testing false positives.

staging
Staging can be updated by ssh'ing into the machine (reading-web-staging-3.eqiad.wmflabs). Always run  before doing anything else so that the logs of what you are doing can be seen by other users.

Updates usually involve either of these commands: cd /srv/mediawiki-vagrant/ vagrant git-update git fetch && git checkout origin/master Sometimes for risky code we may cherry pick changes to staging. Right now we don't have a good process for managing this kind of change. Changes are last in first out so there is always a risk staging is left in an unexpected state.

It may also be necessary to update LocalSettings.php When doing this we should mail the team mailing list.

Browser tests

 * What are browser tests? Quality_Assurance/Browser_testing

Ruby

 * Setup browser tests - Quality_Assurance/Browser_testing/Setup_instructions;
 * Run browser tests - Quality_Assurance/Browser_testing/Running_tests;
 * Have Jenkins execute browser tests for every patchset submitted to Gerrit - Continuous_integration/Browser_tests;
 * Write your own browser tests - Quality_Assurance/Browser_testing/Writing_tests
 * Useful links - Reading/Web/Selenium_tests;
 * Some tips on writing automation tests - Reading/Web/Automation CheatSheet.

Node.js
Selenium/Node.js

Running browser tests on the beta cluster
Set the following variables, using an existing account for username and password: Note that certain tests may require your username to have certain permissions set.

Simulating browser test run on an unmerged patchset.
The way we test such change is to create a copy of the existing Jenkins job, disable all notifications and then fetch and checkout the patchset.

Steps:
 * For example, you want to test 298959
 * Log in to Jenkins
 * Click New Item link in the left sidebar
 * A form would show
 * Item Name: selenium-(repository)-(gerrit-patch-number), so in this example selenium-Wikidata-298959
 * Copy existing Item: selenium-(repository), so in this example selenium-Wikidata
 * Save form
 * Go to the job configuration
 * Click Source Code Management > Git > Advanced
 * Change Refspec from +refs/heads/*:refs/remotes/origin/* to refs/changes/59/298959/1
 * You can get refs/changes... from Gerrit patch
 * Go to the patch 298959
 * click Download
 * URL in Pull will be git pull https://zfilipin@gerrit.wikimedia.org/r/wikidata/browsertests refs/changes/59/298959/1
 * You only need the last section, refs/changes/59/298959/1
 * Change Branch Specifier (blank for 'any') from master to FETCH_HEAD
 * Uncheck Build periodically
 * Delete the sections:
 * Execute shell (only one that starts with # Attempt to figure out MediaWiki branch being used and fetch it out)
 * Editable Email Notification
 * IRC Notification
 * Save and trigger a build of that item-name to run browser tests.

Troubleshooting "flakey tests"
Sometimes our beta cluster integration jobs fail, even though they pass when we merge code and a manual test shows that functionality is working as expected. This is often due to problems with the beta cluster's availability or performance and how the tests are written not the thing they are testing.

Use you can point Selenium jobs to unmerged patches, to test changes to browser tests by cloning an existing job by clicking "New Item" on the dashboard and using "copy from". This results in a duplicate job with a different name e.g. https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue-BugFix/

You can use this job to point run tests from another code path:
 * Click configure
 * Find on page "gerrit". In build "Execute shell" you'll see an explicit checkout command. You can update this line to check out any code for that repository.
 * Run the build (using "Build now")

Current browser test results

 * Dashboard