Reading/Web/QA

We use manual testing as well as automated testing to QA our products.

Our manual testing frequently occurs as part of our sprint process. After code has gone through engineers and design we hand it over for QA.

Manual handover
First it is important to check we have instructions on how and where to test the change in the description, given we have many test environments identifying where testing can take place.

We have 2 environmentsː


 * The beta cluster, which updates frequently as code is merged
 * reading-web-staging.wmflabs.org, which is specialized for testing article content

Depending on what is being tested usually one is more suitable than the other. It's extremely important to clarify where to test to avoid testing false positives.

staging
Staging can be updated by ssh'ing into the machine (readingwebstaging.eqiad.wmflabs). Always run  before doing anything else so that the logs of what you are doing can be seen by other users.

Updates usually involve either of these commands: cd /srv/mediawiki-vagrant/ vagrant git-update git fetch && git checkout origin/master

If this doesn't work, it may be because a change has been cherry picked (see below). In this case, you should either rebase the cherry pick onto master or revert back to master if the change is no longer needed.

cherry picking changes
Sometimes for risky code we may cherry pick changes to staging. Right now we don't have a good process for managing this kind of change other than commenting on the tickets that need these changes.

Changes are last in first out so there is always a risk staging is left in an unexpected state. It is up to the tech-lead to keep track of these changes.

When cherry picking changes, always refer to the "design sign off" and "QA" columns of the kanban board and when done leave a comment.

LocalSettings
It may also be necessary to update LocalSettings.php (either to disable/enable the content provider) or to turn on new feature flags.

On staging, there is a file /vagrant/mediawiki/Staging.php where changes should be added to.

When making a breaking change (e.g. switching language) we should mail the team mailing list. Recently, I've been exploring making use of wgSiteNotice to inform users of the wiki without need for email. You'll find some commented out lines down the bottom of the Staging.php file.

Browser tests

 * What are browser tests? Quality_Assurance/Browser_testing

Ruby

 * Setup browser tests - Quality_Assurance/Browser_testing/Setup_instructions;
 * Run browser tests - Quality_Assurance/Browser_testing/Running_tests;
 * Have Jenkins execute browser tests for every patchset submitted to Gerrit - Continuous_integration/Browser_tests;
 * Write your own browser tests - Quality_Assurance/Browser_testing/Writing_tests
 * Useful links - Reading/Web/Selenium_tests;
 * Some tips on writing automation tests - Reading/Web/Automation CheatSheet.

Node.js
Selenium/Node.js

Running browser tests on the beta cluster
Set the following variables, using an existing account for username and password: Note that certain tests may require your username to have certain permissions set.

Simulating browser test run on an unmerged patchset.
The way we test such change is to create a copy of the existing Jenkins job, disable all notifications and then fetch and checkout the patchset.

Steps:
 * For example, you want to test 298959
 * Log in to Jenkins
 * Click New Item link in the left sidebar
 * A form would show
 * Item Name: selenium-(repository)-(gerrit-patch-number), so in this example selenium-Wikidata-298959
 * Copy existing Item: selenium-(repository), so in this example selenium-Wikidata
 * Save form
 * Go to the job configuration
 * Click Source Code Management > Git > Advanced
 * Change Refspec from +refs/heads/*:refs/remotes/origin/* to refs/changes/59/298959/1
 * You can get refs/changes... from Gerrit patch
 * Go to the patch 298959
 * click Download
 * URL in Pull will be git pull https://zfilipin@gerrit.wikimedia.org/r/wikidata/browsertests refs/changes/59/298959/1
 * You only need the last section, refs/changes/59/298959/1
 * Change Branch Specifier (blank for 'any') from master to FETCH_HEAD
 * Uncheck Build periodically
 * Delete the sections:
 * Execute shell (only one that starts with # Attempt to figure out MediaWiki branch being used and fetch it out)
 * Editable Email Notification
 * IRC Notification
 * Save and trigger a build of that item-name to run browser tests.

Troubleshooting "flakey tests"
Sometimes our beta cluster integration jobs fail, even though they pass when we merge code and a manual test shows that functionality is working as expected. This is often due to problems with the beta cluster's availability or performance and how the tests are written not the thing they are testing.

Use you can point Selenium jobs to unmerged patches, to test changes to browser tests by cloning an existing job by clicking "New Item" on the dashboard and using "copy from". This results in a duplicate job with a different name e.g. https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue-BugFix/

You can use this job to point run tests from another code path:
 * Click configure
 * Find on page "gerrit". In build "Execute shell" you'll see an explicit checkout command. You can update this line to check out any code for that repository.
 * Run the build (using "Build now")

Current browser test results

 * Dashboard