Readers/Web/Instrumentation Overview

Prerequisites

 * Get research objective and schema from data analyst, both of which can be found in the Phabricator ticket.
 * Name of what is being tested: Identify the components that will be tested. In this example, we are A/B testing the Zebra skin, so the name is skin-vector-zebra-experiment
 * Variations: Identify the different versions of the component that you'll be testing. The variations in this example are vector-feature-zebra-design-enabled and vector-feature-zebra-design-disabled

Tools Used

 * 1) If setting the test in the Vector skin, utilize the Built-In Vector Feature Manager
 * 2) Hue - Use this to query events. (Remember: Data is limited to 90 days)
 * 3) DataHub Historically, the team has been using Google Spreadsheets for schema tracking, but we are currently transitioning to recording schemas here.

Writing the variations

 * 1) Configure testing parameters in LocalSettings.php, such as the percentage of traffic that will see each version.
 * 2) Define an array for A/B testing in the Vector skin of MediaWiki.

To, allocate 50% of users to the "control" bucket and 50% to the "treatment" bucket, use the following format.

Launching Test
Once you've set up your A/B test and determined your sample size, commit the patch containing the test as seen here.

Analyst Handoff
After the test has run for a sufficient amount of time, the analyst will check the results to determine which variation performed better.

Next Steps

 * Once we have identified the winning variation and project manager approves, open a new patch to implement it. (Example)
 * Ensure that the changes are properly documented and communicated to relevant stakeholders.