Extension:GrowthExperiments/developer setup

From mediawiki.org


git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/GrowthExperiments" extensions/GrowthExperiments
git clone "https://gerrit.wikimedia.org/r/mediawiki/skins/Vector" skins/Vector
git clone "https://gerrit.wikimedia.org/r/mediawiki/skins/MinervaNeue" skins/MinervaNeue
git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/CirrusSearch" extensions/CirrusSearch
git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/Echo" extensions/Echo
git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/Elastica" extensions/Elastica
git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/MobileFrontend" extensions/MobileFrontend
git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/PageViewInfo" extensions/PageViewInfo
git clone --recurse-submodules "https://gerrit.wikimedia.org/r/mediawiki/extensions/VisualEditor" extensions/VisualEditor
  • Download and place the file(s) in a directory called GrowthExperiments in your extensions/ folder.
  • Add the following code at the bottom of your LocalSettings.php :
    wfLoadExtension( 'GrowthExperiments' );
    wfLoadSkin( 'Vector' );
    wfLoadSkin( 'MinervaNeue' );
    wfLoadExtension( 'CirrusSearch' );
    wfLoadExtension( 'Echo' );
    wfLoadExtension( 'Elastica' );
    wfLoadExtension( 'PageViewInfo' );
    wfLoadExtension( 'MobileFrontend' );
    wfLoadExtension( 'VisualEditor' );
  • Run the update script which will automatically create the necessary database tables that this extension needs.
  • Follow additional CirrusSearch instructions (i.e. installing composer-based dependencies for Cirrus and Elastic).
  • Most of the above require no configuration and work out-of-the-box (troubleshooting: Elastica, PageViewInfo, MobileFrontend, VisualEditor)
  • Yes Done – Navigate to Special:Version on your wiki to verify that the extension is successfully installed.

Vagrant installation:

  • If using Vagrant , install with vagrant roles enable growthexperiments addlink --provision

Other extensions (optional)

Enabling Growth features[edit]

  1. Go to Special:Preferences (while logged in)
  2. Under User profile section: check Display newcomer homepage, Default to newcomer homepage from username link and Enable the editor help panel

Alternatively, run the following in the browser console:

new mw.Api().saveOptions({
	'growthexperiments-help-panel-tog-help-panel': '1',
	'growthexperiments-homepage-enable': '1',
	'growthexperiments-homepage-pt-link': '1',


Mentor dashboard[edit]

To enable the mentor dashboard follow the next steps:

  1. Log in as an admin
  2. Create a new page for instance Mentors_list to hold the list of users that will be mentors, add a mentor by adding * User:MentorUsername|Description. More info https://www.mediawiki.org/wiki/Growth/Communities/How_to_configure_the_mentors%27_list#create
  3. Go to http://localhost:8080/wiki/Special:EditGrowthConfig
  4. Under mentorship, enable mentorship & specify the page containing the list of mentors: Mentors_list.
  5. On your MediaWiki installation and Run php extensions/GrowthExperiments/maintenance/updateMenteeData.php script
  6. Log in to the mentor account & go to http://localhost:8080/wiki/Special:MentorDashboard

Suggested Edits[edit]

There are three basic approaches for setting up suggested edits, the main functionality of GrowthExperiments, in a developer setup:

  1. Have the extension use the search API of a remote (production) wiki. You won't have to deal with setting up search locally, and you will get a wide range of realistic task suggestions, but the articles suggested for those tasks won't exist on your wiki so most editing-related functionality won't work. This is the easiest way to get the homepage and guidance working in general, and to QA task suggestions. It doesn't work for structured link recommendations though since those aren't enabled on any production wiki yet.
  2. Set up search locally, copy-paste or import a couple articles from some real wiki. This is quite a bit more effort but doable (on Vagrant it should mostly work out of the box). It's good for backend development, probably not really worth the effort for frontend development.
  3. Mock all the backend logic involved with static PHP code. This is nice for frontend work as you get direct and full control over the responses from the backend.

Use remote search[edit]

use GrowthExperiments\AqsEditInfoService;
use MediaWiki\MediaWikiServices;

# Use the test config for en.wikipedia.org
$wgGENewcomerTasksConfigTitle = 'mw:Growth/Personalized_first_day/Newcomer_tasks/Prototype/templates/en.json';

# Search for tasks on en.wikipedia.org
$wgGENewcomerTasksRemoteApiUrl = 'https://en.wikipedia.org/w/api.php';

# Get extra data for the task cards from en.wikipedia.org
$wgGERestbaseUrl = 'https://en.wikipedia.org/api/rest_v1';
$wgHooks['MediaWikiServices'][] = function ( MediaWikiServices $services ) {
	$services->redefineService( 'GrowthExperimentsEditInfoService', function ( MediaWikiServices $services ) {
		return new AqsEditInfoService( $services->getHttpRequestFactory(), 'en.wikipedia' );
	} );

Set up search locally[edit]

In Vagrant search can be set up with vagrant roles enable cirrussearch --provision (but it will be set up automatically as a dependency if you provision the growthexperiments role). For Docker, see MediaWiki-Docker/Configuration recipes/ElasticSearch. For other setups, a basic configuration is:

wfLoadExtension( 'CirrusSearch' );
require_once "$IP/extensions/CirrusSearch/tests/jenkins/FullyFeaturedConfig.php";
$wgCirrusSearchServers = [ 'elasticsearch' ];
$wgCirrusSearchWMFExtraFeatures = [
	'weighted_tags' => [ 'build' => true, 'use' => true ]

See the setup notes of Extension:CirrusSearch for more details.

You can use the CirrusSearch maintenance script UpdateWeightedTags.php for setting the ORES topics of wiki pages.

To use settings from cswiki:

  1. Follow the ElasticSearch instructions from MediaWiki-Docker/Configuration_recipes/ElasticSearch.
  2. Copy the contents of MediaWiki:NewcomerTopicsOres.json to http://localhost:8080/w/index.php?title=MediaWiki:NewcomerTopicsOres.json&action=edit. (must be signed in as an admin on MediaWiki)
  3. Copy the contents of MediaWiki:NewcomerTasks.json to http://localhost:8080/w/index.php?title=MediaWiki:NewcomerTasks.json&action=edit. (must be signed in as an admin on MediaWiki)
  4. Add $wgGENewcomerTasksTopicType = 'ores' to LocalSettings.php.
  5. Add $wgGERestbaseUrl = 'https://cs.wikipedia.org/api/rest_v1'; to LocalSettings.php.
  6. Add $wgGENewcomerTasksGuidanceEnabled = true; to LocalSettings.php.

To use settings from other wikis, replace "cs" with the language code of the desired wiki (for MediaWiki:NewcomerTasks.json and $wgGERestbaseUrl).

Mock the backend[edit]

use GrowthExperiments\NewcomerTasks\AddImage\SubpageImageRecommendationProvider;
use GrowthExperiments\NewcomerTasks\AddLink\SubpageLinkRecommendationProvider;
use GrowthExperiments\NewcomerTasks\ConfigurationLoader\StaticConfigurationLoader;
use GrowthExperiments\NewcomerTasks\TaskSuggester\StaticTaskSuggesterFactory;
use GrowthExperiments\NewcomerTasks\TaskSuggester\TaskSuggesterFactory;
use GrowthExperiments\NewcomerTasks\TaskType\LinkRecommendationTaskType;
use GrowthExperiments\NewcomerTasks\Task\Task;
use GrowthExperiments\NewcomerTasks\TaskType\TaskType;
use MediaWiki\MediaWikiServices;

# Enable under-development features still behind feature flag:
$wgGENewcomerTasksLinkRecommendationsEnabled = true;
$wgGELinkRecommendationsFrontendEnabled = true;

$wgHooks['MediaWikiServices'][] = function ( MediaWikiServices $services ) {
	$linkRecommendationTaskType = new LinkRecommendationTaskType( 'link-recommendation', TaskType::DIFFICULTY_EASY, [] );

	# Mock the configuration, which would normally be at MediaWiki:NewcomerTaskConfig.json, to have just one 'link-recommendation' task type.
	$services->redefineService( 'GrowthExperimentsNewcomerTasksConfigurationLoader', function ( MediaWikiServices $services ) use ( $linkRecommendationTaskType ) {
		return new StaticConfigurationLoader( [ $linkRecommendationTaskType ] );
	} );

	# Mock the task suggester to specify what article(s) will be suggested.
	$services->redefineService( 'GrowthExperimentsTaskSuggesterFactory', function ( MediaWikiServices $services ) use ( $linkRecommendationTaskType ): TaskSuggesterFactory {
		return new StaticTaskSuggesterFactory( [
			new Task( $linkRecommendationTaskType, new TitleValue( NS_MAIN, 'Douglas Adams' ) ),
		] );
	} );

# Set up SubpageLinkRecommendationProvider, which will take the recommendation from the article's /addlink.json subpage,
# e.g. [[Douglas Adams/addlink.json]]. The output of https://addlink-simple.toolforge.org can be copied there.
$wgHooks['MediaWikiServices'][] = SubpageLinkRecommendationProvider::class . '::onMediaWikiServices';
$wgHooks['ContentHandlerDefaultModelFor'][] = SubpageLinkRecommendationProvider::class . '::onContentHandlerDefaultModelFor';
# Same for image recommendations, with addimage.json and http://image-suggestion-api.wmcloud.org/?doc
$wgHooks['MediaWikiServices'][] = SubpageImageRecommendationProvider::class . '::onMediaWikiServices';
$wgHooks['ContentHandlerDefaultModelFor'][] = SubpageImageRecommendationProvider::class . '::onContentHandlerDefaultModelFor';

Link recommendations[edit]

Your local development environment can query the "external traffic" production release of the link recommendation service. Access is proxied via the api-gateway (api.wikimedia.org) which requires an access token to utilize the API. That means you need to generate a personal access token (https://api.wikimedia.org/wiki/Documentation/Getting_started/Authentication#Personal_API_tokens), when you will use the access token as the value for GELinkRecommendationServiceAccessToken in LocalSettings.php:

$wgGELinkRecommendationServiceWikiIdMasquerade = 'cswiki';
$wgGELinkRecommendationServiceAccessToken = "#access token from api.wikimedia.org";
$wgGELinkRecommendationServiceUrl = 'https://api.wikimedia.org/service/linkrecommendation';

// Optionally set the fallback on DB miss variable. In practice, if you are
// using refreshLinkRecommendations.php and importOresTopics.php maintenance
// scripts locally, then you probably want this config flag set to false (default)
// $wgGELinkRecommendationFallbackOnDBMiss = true;

To allow non-existent pages (red links) as suggested edits, add the following to LocalSettings.php

$wgGEDeveloperSetup = true;

To enable link recommendation tasks for the currently logged in user, run the following from the browser console (while on Special:Homepage):


To update the link recommendations for each topic:

php /path/to/mediawiki/extensions/GrowthExperiments/maintenance/refreshLinkRecommendations.php

To see which articles have corresponding link recommendation tasks:

  1. Go to your local wiki
  2. Search for hasrecommendation:link

Image recommendations[edit]

API: 2022 production edition[edit]

You need to set up an SSH tunnel as there is not a public facing API yet.

  1. edit /etc/hosts on your laptop and add image-suggestion.discovery.wmnet
  2. set up your SSH tunnel: ssh -N mwmaint1002.eqiad.wmnet -L 30443:image-suggestion.discovery.wmnet:30443
  3. Make a test request on your laptop: curl -k https://image-suggestion.discovery.wmnet:30443/public/image_suggestions/suggestions/ptwiki/6721577 Note that the -k option is needed to ignore SSL validation

To skip SSL validation, make sure the following is set:

$wgGEDeveloperSetup = true;

The image suggestions API retrieves suggestions for a given article ID in production. In non-production environments, we need to query by the article title rather than the ID since the article ID is different. This is controlled by GEImageRecommendationServiceUseTitles (should be set to true by default).

Finally, set the config values in your LocalSettings.php:

$wgGEImageRecommendationServiceUrl = 'https://image-suggestion.discovery.wmnet:30443';
$wgGEImageRecommendationApiHandler = "production";
// ID of the wiki from which to query suggestions
$wgGEImageRecommendationServiceWikiIdMasquerade = "ptwiki";

API: MVP edition[edit]

To use the live image suggestions API with the Wikipedia matching the wiki's language, set

$wgGEImageRecommendationServiceUrl = 'https://image-suggestion-api.wmcloud.org';
$wgGEImageRecommendationApiHandler = 'mvp';

Other settings

To use Commons as a file repository (without this the image metadata won't work), set

$wgUseInstantCommons = true;

To enable image recommendation tasks for the currently logged in user, run the following from the browser console (while on Special:Homepage):


To tag an article for image recommendation task:

php /path/to/mediawiki/extensions/CirrusSearch/maintenance/UpdateWeightedTags.php --tagType 'recommendation.image' --page $articleTitle
php /path/to/mediawiki/maintenance/runJobs.php

To see which articles have corresponding image recommendation tasks:

  1. Go to your local wiki
  2. Search for hasrecommendation:image

Seeding Articles from Czech Wikipedia[edit]

See also https://phabricator.wikimedia.org/T274198#6972115

  1. Make sure that http://localhost:8080/wiki/MediaWiki:NewcomerTasks.json is the same as that in Czech Wikipedia.
  2. Get article titles for Growth tasks from Czech Wikipedia.
    curl -s "https://cs.wikipedia.org/w/api.php?action=query&format=json&export=1&exportnowrap=1&exportschema=0.11&generator=growthtasks" > tasks.xml
  3. Import XML dump by running the following command inside mediawiki installation directory
    php maintenance/importDump.php --report=10 tasks.xml
  4. Create "Module:Wikidata" on your local site. Copy/paste the source of https://cs.wikipedia.org/wiki/Module:Wikidata and save it.
  5. Update secondary tables (to get accurate information on Special:RecentChanges and Special:Statistics)
    php maintenance/rebuildrecentchanges.php
    php maintenance/initSiteStats.php
  6. Update ElasticSearch
    php maintenance/update.php --quick
    php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php && \
    php extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip && \
    php extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse && \
    php maintenance/runJobs.php

User impact[edit]

(this is work in progress and will change)

By default, user impact functionality is configured to use a subpage provider, ie. for every user impact data will be read from User:<username>/userimpact.json, in the format used by [Expensive]UserImpact::jsonSerialize().

To use real data, you can do something like

use GrowthExperiments\UserImpact\UserImpactLookup;
use MediaWiki\MediaWikiServices;

$wgHooks['MediaWikiServices'][] = function ( MediaWikiServices $services ) {
    $services->redefineService( 'GrowthExperimentsUserImpactLookup',
        function ( MediaWikiServices $services ): UserImpactLookup {
            return $services->get( '_GrowthExperimentsUserImpactLookup_Computed' );
} );

Seeding ORES topics[edit]

To import ORES topics for articles imported from a production wiki (English Wikipedia in the example below), run

php extensions/GrowthExperiments/maintenance/importOresTopics.php --apiUrl https://en.wikipedia.org/w/api.php --wikiId enwiki --count 100 --verbose

To set ORES topics for some article manually, use

php extensions/CirrusSearch/maintenance/UpdateWeightedTags.php --tagType classification.ores.articletopic --tagName 'Regions.Americas.Central America' --weight 750 --tagName 'Media.Media*' --weight 800 --page={yourPageTitle}