MediaWiki-Docker/Configuration recipes/TestKitchen

Minimal Setup

If you only need to satisfy a dependency on TestKitchen (likely the ext.testKitchen ResourceLoader module), then all you need to do is:

Clone the TestKitchen extension and its dependencies into the extensions directory:

for extension in EventStreamConfig EventLogging; do
  git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/${extension}"
done

Add the following to LocalSettings.php:

wfLoadExtensions( [
    'EventStreamConfig',
    'EventLogging',
    'TestKitchen'
] );

// Causes mw.testKitchen ResourceLoader module to be sent to the browser
$wgTestKitchenEnableExperiments = true;

TestKitchen Lite Setup

This section will guide you through setting up TestKitchen without Test Kitchen UI. You will still be able to test your experiments but your workflow won't be the same as it would be in production.

Project Setup

Extensions

You need to install these extensions:

To install these extensions, run the following command from the root folder of your MediaWiki installation:

for extension in EventStreamConfig EventLogging TestKitchen; do
  git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/${extension}"
done

Configuration

Add the following to the LocalSettings.php file:

<?php

wfLoadExtensions( [
	'EventStreamConfig',
	'EventLogging',
	'TestKitchen',
] );

// EventStreamConfig Configuration
// ===============================

// The Test Kitchen SDK uses its own Metrics Platform Client (MPC) instances internally.
// These MPCs are separately configured and only contain configuration for the
// product_metrics.web_base event stream.
$wgEventStreams = [
	'product_metrics.web_base' => [
		'schema_title' => 'analytics/product_metrics/web/base',
		'producers' => [
			'metrics_platform_client' => [
				'provide_values' => [

					// Make platform (e.g. desktop or mobile) available as
					// a dimension during analysis.
					'agent_client_platform',
					'agent_client_platform_family',

					// Make user authentication status available as a dimension
					// during analysis.
					'performer_is_logged_in',
					'performer_is_temp',
					
					// Enable the calculation of the "click-through per page
					// visit" generic metric.
					'performer_pageview_id',

					// The ClickThroughRateInstrument instrument uses this
					// stream by default. Capture the "smart session ID"
					// contextual attribute so that analysts can calculate all
					// three flavors of click-through rate.
					'performer_active_browsing_session_token',
				],
			],
		],
	],
];

$wgEventLoggingStreamNames = array_keys( $wgEventStreams );

// TestKitchen Configuration
// =============================

// Causes mw.testKitchen ResourceLoader module to be sent to the browser
$wgTestKitchenEnableExperiments = true;

// Enable the user to override experiment enrollments using
// mw.testKitchen.overrideExperimentGroup() / mw.testKitchen.clearExperimentOverride()
$wgTestKitchenEnableExperimentOverrides = true;

Testing Experiments

For a more detailed walkthrough of testing and debugging an experiment, see https://wikitech.wikimedia.org/wiki/TestKitchen/Conduct_an_experiment#Test_and_debug.

Testing an Experiment on Logged-In Users

You can run a experiment on logged-in users by adding something like the following to LocalSettings.php and reloading the page:

$wgTestKitchenExperiments = [

    // A simple logged-in user experiment. 50% of logged-in users will be
    // enrolled in the experiment and assigned either the control or treatment
    // group.
    [
        'name' => 'my-awesome-experiment',
        'sample' => [
            'rate' => 0.5, // 50% of all logged-in users
        ],
        'groups' => [
            'control',
            'treatment'
        ],
        'user_identifier_type' => 'mw-user',
        'stream_name' => 'product_metrics.web_base',
        'contextual_attributes' => [
            'page_id',
            'page_title'
        ]
    ],
];

And running the following in your browser's console:

const e = mw.testKitchen.getExperiment( 'my-awesome-experiment' );

e.send( 'Hello, World!' );

If you weren't enrolled in the experiment or were assigned the wrong group, then you can override your experiment enrollment with mw.testKitchen.overrideExperimentGroup(), e.g.

mw.testKitchen.overrideExperimentGroup( 'my-awesome-experiment', 'treatment' );

Testing an Experiment on All Traffic

MediaWiki assumes the X-Experiment-Enrollments header was set by Varnish. Therefore, "running" an experiment on all traffic is as simple as setting the header on every request using a browser extension.

Set the X-Experiment-Enrollments header to my-awesome-experiment:control;, reload the page, and then run the following in your browser's console:

const e = mw.testKitchen.getExperiment( 'my-awesome-experiment' );

e.send( 'Hello, World!' );

TestKitchen Setup

Project Setup

Extensions

You need to install these extensions:

To install these extensions, run the following command from the root folder of your MediaWiki installation:

for extension in EventStreamConfig EventLogging TestKitchen; do
  git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/${extension}"
done

Applications

You need to install and run the Test Kitchen UI application. To do this, simply define two new services in the docker-compose.override.yml file in the root folder of your MediaWiki installation:

services:
  test-kitchen:
    extends:
      file: ./extensions/TestKitchen/devserver/docker-compose.yml
      service: testkitchen

  test-kitchen-db:
    extends:
      file: ./extensions/TestKitchen/devserver/docker-compose.yml
      service: test-kitchen-db
    volumes:
      - ./cache/test-kitchen-db:/var/lib/mysql

And then run the following commands:

# if updating and suspecting db errors, rm -rf ./cache/test-kitchen-db/* might help
mkdir ./cache/test-kitchen-db
docker compose up -d

Configuration

Add the following to the LocalSettings.php file:

<?php

wfLoadExtensions( [
	'EventStreamConfig',
	'EventLogging',
	'TestKitchen',
] );

// EventStreamConfig Configuration
// ===============================

// The Test Kitchen SDK uses its own Metrics Platform Client (MPC) instances internally.
// These MPCs are separately configured and only contain configuration for the
// product_metrics.web_base event stream.
$wgEventStreams = [
	'product_metrics.web_base' => [
		'schema_title' => 'analytics/product_metrics/web/base',
		'producers' => [
			'metrics_platform_client' => [
				'provide_values' => [

					// Make platform (e.g. desktop or mobile) available as
					// a dimension during analysis.
					'agent_client_platform',
					'agent_client_platform_family',

					// Make user authentication status available as a dimension
					// during analysis.
					'performer_is_logged_in',
					'performer_is_temp',
					
					// Enable the calculation of the "click-through per page
					// visit" generic metric.
					'performer_pageview_id',

					// The ClickThroughRateInstrument instrument uses this
					// stream by default. Capture the "smart session ID"
					// contextual attribute so that analysts can calculate all
					// three flavors of click-through rate.
					'performer_active_browsing_session_token',
				],
			],
		],
	],
];

$wgEventLoggingStreamNames = array_keys( $wgEventStreams );

// TestKitchen Configuration
// =============================

// Causes mw.testKitchen ResourceLoader module to be sent to the browser
$wgTestKitchenEnableExperiments = true;

// Enable the user to override experiment enrollments using
// mw.testKitchen.overrideExperimentGroup() / mw.testKitchen.clearExperimentOverride()
$wgTestKitchenEnableExperimentOverrides = true;

// Enable fetching logged-in experiment configs from TestKitchen UI.
$wgTestKitchenInstrumentConfiguratorBaseUrl = 'http://test-kitchen:8086';
$wgTestKitchenEnableExperimentConfigsFetching = true;