Readers/Web/Instrumentation Overview

2 Months from Launch

 * Gather prerequisites
 * Plan in advance what to do with results and what actions to take.

1.5 Months from Launch
Begin writing instrumentation

1 Month from Launch
Complete A/B Test instrumentation

2 Weeks from Launch
Enable a dummy A/B test earlier to test the mechanism separately from the actual test in Beta Cluster and Test Wiki (production).

1 Week from Launch
Deploy on smaller language wikis and test

Launch Date
Deploy on English wikipedia

Two Weeks after Launch
Turn off A/B test

Prerequisites

 * Get research objective and schema from data analyst, both of which can be found in the Phabricator ticket.
 * Name of what is being tested: Identify the components that will be tested. In this example, we are A/B testing the Zebra skin, so the name is skin-vector-zebra-experiment. Keep this for later.
 * Variations: Identify the different versions of the component that you'll be testing. The variations in this example are vector-feature-zebra-design-enabled and vector-feature-zebra-design-disabled.
 * Make sure the answer is "yes" for the following question: "Is the focus or the aim of the test on change management and phasing out features to users?"

Server Side
skin.json


 * 1) Modify the lines containing the A/B test configuration.
 * 2) In "VectorWebABTestEnrollment", assign "name": to the value decided on earlier (skin-vector-zebra-experiment).

ABRequirement.php

This file checks if a user is part of a specific A/B test experiment and determines whether they should be in the "control" group or the "test" group based on their user ID. Currently, it is hardcoded to divide users into two groups.

ServiceWiring.php This PHP file defines a set of service wirings for the Vector skin used in MediaWiki core. The purpose of these wirings is to manage different features and requirements for the Vector skin. The file includes:


 * 1) A main function that starts with a   statement, which indicates that this file returns an array of service definitions.
 * 2) An array containing a single key-value pair, where the key is a constant  representing the service name, and the value is an anonymous function that creates and configures the   object.
 * 3) Inside the anonymous function:
 * 4) * A new instance of  is created, which will manage the registration and evaluation of different features.
 * 5) * Several "requirements" are registered with the . These requirements define the conditions that must be met for a feature to be enabled for a particular user.

Example: Zebra Design Feature

This feature depends on the Zebra AB test and whether the Zebra design configuration is enabled. ⬇️

The following registers a feature named  with the. To enable this feature, three requirements must be met: the skin must be fully initialized, the  condition must be satisfied, and the   condition must also be fulfilled. The new feature (in this case the Zebra Design Feature) consumes ABRequirement as a requirement.

Zebra is enabled when:


 * 1) Zebra config is enabled
 * 2) Zebra AB Test config is disabled
 * 3) Zebra AB Test config is enabled (50% chance)

FeatureManager.php

The  class in this file provides a way to manage features and requirements for the Vector skin. It allows for decoupling the logic of different components from their requirements, making the code more flexible and maintainable

The below method returns a list of CSS classes that should be added to the  tag of the skin based on the enabled features. It iterates through the registered features and checks if each one is enabled or disabled. Based on the result, it generates CSS classes to be added to the body tag for styling purposes. In this case  or

Client Side
LESS files

TO-DO: Fix the issues with the .vector-body class and use the stable .mw-body-content class instead. Or explore a different approach to using the feature flag that reduces the risk of specificity-related bugs.

Other useful tools

 * 1) Hue - Use this to query events. (Remember: Data is limited to 90 days)
 * 2) DataHub Historically, the team has been using Google Spreadsheets for schema tracking, but we are currently transitioning to referencing and recording schemas here.

Writing the variations

 * 1) Configure testing parameters in LocalSettings.php, such as the percentage of traffic that will see each version.
 * 2) Define an array for A/B testing in the Vector skin of MediaWiki.

To allocate 50% of users to the "control" bucket and 50% to the "treatment" bucket, use the following format.

Launching test
Once you've set up your A/B test and determined your sample size, commit the patch containing the test as seen here.

Coordinate with the PM and CRS and ensure they are aware of the test schedule.

Most of the time, the integrity of the test means there won't be a public announcement ahead of time.


 * Otherwise, the team can coordinate to ensure that the messages announcing the test have been posted.


 * This would happen at least a week before the estimated time of launch – to give heads-up about a change of user experience, and make it possible for the communities to look for possible bugs in the user-generated code.
 * Note that the quality of the configuration (wmf-config/InitialiseSettings.php) should be confirmed before these steps.

Phase 1
Limit configuration enabling to test.wikipedia.org and test2.wikipedia.org. Avoid launching the test on active content wikis without launching it on test wikis or closed content wikis first.

Analyst handoff
After the test has run for a sufficient amount of time, the analyst will check the results to determine which variation performed better.

Next steps

 * Once we have identified the winning variation and project manager approves, open a new patch to implement it. (Example forthcoming)
 * Ensure that the changes are properly documented and communicated to relevant stakeholders.