Readers/Web/Instrumentation Overview

2 Months from Launch

 * Gather prerequisites
 * Plan in advance what to do with results and what actions to take.

1.5 Months from Launch
Begin writing instrumentation

1 Month from Launch
Complete A/B Test instrumentation

2 Weeks from Launch
Enable a dummy A/B test earlier to test the mechanism separately from the actual test in Beta Cluster and Test Wiki (production).

1 Week from Launch
Deploy on smaller language wikis and test

Launch Date
Deploy on English wikipedia

Two Weeks after Launch
Turn off A/B test

Prerequisites

 * Get research objective and schema from data analyst, both of which can be found in the Phabricator ticket.
 * Name of what is being tested: Identify the components that will be tested. In this example, we are A/B testing the Zebra skin, so the name is skin-vector-zebra-experiment. Keep this for later.
 * Variations: Identify the different versions of the component that you'll be testing. The variations in this example are vector-feature-zebra-design-enabled and vector-feature-zebra-design-disabled.
 * Make sure the answer is "yes" for the following question: "Is the focus or the aim of the test on change management and phasing out features to users?"

Server Side
skin.json


 * 1) Modify the lines containing the A/B test configuration.
 * 2) In "VectorWebABTestEnrollment", assign "name": to the value decided on earlier (skin-vector-zebra-experiment).

ABRequirement.php

This file checks if a user is part of a specific A/B test experiment and determines whether they should be in the "control" group or the "test" group based on their user ID. Currently, it is hardcoded to divide users into two groups.

ServiceWiring.php This PHP file defines a set of service wirings for the Vector skin used in MediaWiki core. The purpose of these wirings is to manage different features and requirements for the Vector skin. The file includes:


 * 1) A main function that starts with a   statement, which indicates that this file returns an array of service definitions.
 * 2) An array containing a single key-value pair, where the key is a constant  representing the service name, and the value is an anonymous function that creates and configures the   object.
 * 3) Inside the anonymous function:
 * 4) * A new instance of  is created, which will manage the registration and evaluation of different features.
 * 5) * Several "requirements" are registered with the . These requirements define the conditions that must be met for a feature to be enabled for a particular user.

Example: Zebra Design Feature

This feature depends on the Zebra AB test and whether the Zebra design configuration is enabled. ⬇️

The following registers a feature named  with the. To enable this feature, three requirements must be met: the skin must be fully initialized, the  condition must be satisfied, and the   condition must also be fulfilled. The new feature (in this case the Zebra Design Feature) consumes ABRequirement as a requirement.

Zebra is enabled when:


 * 1) Zebra config is enabled
 * 2) Zebra AB Test config is disabled
 * 3) Zebra AB Test config is enabled (50% chance)

FeatureManager.php

The  class in this file provides a way to manage features and requirements for the Vector skin. It allows for decoupling the logic of different components from their requirements, making the code more flexible and maintainable

The below method returns a list of CSS classes that should be added to the  tag of the skin based on the enabled features. It iterates through the registered features and checks if each one is enabled or disabled. Based on the result, it generates CSS classes to be added to the body tag for styling purposes. In this case  or

Client Side
skin.js

The  file contains JavaScript code to initialize and manage various functionalities of the Vector skin, including language buttons, toggles, menus, search, animations, and A/B testing.

The script calls the  function first to initialize the skin. Then, it checks if A/B tests are enabled and the user is not anonymous. If A/B tests are enabled for the user, it initializes A/B tests using the  function with the configuration provided in. ⬇️

AB.js

The  handles A/B testing functionality for web experiments. It exports a function called, which is used to initialize and manage A/B tests.

Types and Definitions:


 * : A function that takes an optional string parameter and returns a boolean.
 * : An object representing an A/B test with properties such as name and various functions for testing.
 * : An object representing the desired sampling rate for a group in the range [0, 1].
 * : An object representing the properties needed to define an A/B test, such as experiment name, buckets, and token.

Function:

This function is the main entry point of the module. It takes the following parameters:


 * 1)   (WebABTestProps): An object containing the properties of the A/B test, such as experiment name, buckets, and token.
 * 2)   (string): A unique token that identifies the subject (user) for the duration of the experiment.
 * 3)   (boolean, optional): A flag to force the initialization of the A/B test event. This is used for testing purposes and bypasses caching.

The function returns a  object, which encapsulates the A/B test and provides various methods to check the subject's bucket, sample status, and treatment bucket assignment.

Bucketing Mechanism:

The  function uses a bucketing mechanism to assign users to different buckets based on the sampling rates defined in the   object. The buckets represent different variations or treatments of the experiment.

If the bucketing has already occurred on the server-side (e.g., by adding a class to the body tag with the bucket name), the function retrieves the bucket from the DOM. Otherwise, it uses the provided token to bucket the subject on the client-side using  function (see next file sample).

Methods of :


 * : Returns the name of the bucket the subject is assigned to for the A/B test.
 * : Checks if the subject is in a specific target bucket.
 * : Determines if the subject is included in the A/B test (i.e., not excluded).
 * : Checks if the subject is in a treatment bucket based on a case-insensitive substring check in the bucket name.

Initialization and Hook:

The A/B test enrollment is logged using a hook and sent to WikimediaEvents if the subject has been sampled into the experiment. Initialization occurs when the  function is called, and it can be forced using the   parameter for testing purposes.

core/mediawiki.experiments.js


 * 1) The module has a function called , which sets up an A/B test experiment.
 * 2) The experiment has different "buckets" to assign users. Each bucket has a certain chance of being chosen.
 * 3) When a user enters the experiment, the system generates a "hash" based on the user's identity and the experiment's name. This hash determines which bucket the user is put into.
 * 4) The user is then shown the content or feature corresponding to their bucket.
 * 5) The experiment can be enabled or disabled, and if it's disabled, all users will be put in a default "control" bucket.

⬇️

modules/ext.wikimediaEvents/webABTestEnrollment.js

This file is part of the WikimediaEvents extension. It is used to log the enrollment of users into A/B tests.

logs the A/B test initialization event with relevant data like the user's group, the experiment name, whether the user is anonymous, etc.

RIC


 * On page load, it checks whether to log the A/B test initialization by waiting for the browser to be idle using.
 * When the A/B test enrollment data is available through a hook, it calls the  function to log the relevant data.

LESS files

TO-DO: Fix the issues with the .vector-body class and use the stable .mw-body-content class instead. Or explore a different approach to using the feature flag that reduces the risk of specificity-related bugs.

Other useful tools

 * 1) Hue - Use this to query events. (Remember: Data is limited to 90 days)
 * 2) DataHub Historically, the team has been using Google Spreadsheets for schema tracking, but we are currently transitioning to referencing and recording schemas here.

Writing the variations

 * 1) Configure testing parameters in LocalSettings.php, such as the percentage of traffic that will see each version.
 * 2) Define an array for A/B testing in the Vector skin of MediaWiki.

To allocate 50% of users to the "control" bucket and 50% to the "treatment" bucket, use the following format.

Launching test
Once you've set up your A/B test and determined your sample size, commit the patch containing the test as seen here.

Coordinate with the PM and CRS and ensure they are aware of the test schedule.

Most of the time, the integrity of the test means there won't be a public announcement ahead of time.


 * Otherwise, the team can coordinate to ensure that the messages announcing the test have been posted.


 * This would happen at least a week before the estimated time of launch – to give heads-up about a change of user experience, and make it possible for the communities to look for possible bugs in the user-generated code.
 * Note that the quality of the configuration (wmf-config/InitialiseSettings.php) should be confirmed before these steps.

Phase 1
Limit configuration enabling to test.wikipedia.org and test2.wikipedia.org. Avoid launching the test on active content wikis without launching it on test wikis or closed content wikis first.

Analyst handoff
After the test has run for a sufficient amount of time, the analyst will check the results to determine which variation performed better.

Next steps

 * Once we have identified the winning variation and project manager approves, open a new patch to implement it. (Example forthcoming)
 * Ensure that the changes are properly documented and communicated to relevant stakeholders.