Extension:GrowthExperiments/developer setup

Installation

Download MediaWiki and set it up with either MediaWiki-Docker or a local server using the manual installation instructions.
Download and install the skins and extensions GrowthExperiments depends on:

git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/GrowthExperiments" extensions/GrowthExperiments
git clone "https://gerrit.wikimedia.org/r/mediawiki/skins/Vector" skins/Vector
git clone "https://gerrit.wikimedia.org/r/mediawiki/skins/MinervaNeue" skins/MinervaNeue
git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/CirrusSearch" extensions/CirrusSearch
git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/Echo" extensions/Echo
git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/Elastica" extensions/Elastica
git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/MobileFrontend" extensions/MobileFrontend
git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/PageViewInfo" extensions/PageViewInfo
git clone --recurse-submodules "https://gerrit.wikimedia.org/r/mediawiki/extensions/VisualEditor" extensions/VisualEditor

Download and place the file(s) in a directory called GrowthExperiments in your extensions/ folder.

Add the following code at the bottom of your LocalSettings.php file:

wfLoadExtension( 'GrowthExperiments' );
wfLoadSkin( 'Vector' );
wfLoadSkin( 'MinervaNeue' );
wfLoadExtension( 'CirrusSearch' );
wfLoadExtension( 'Echo' );
wfLoadExtension( 'Elastica' );
wfLoadExtension( 'PageViewInfo' );
wfLoadExtension( 'MobileFrontend' );
wfLoadExtension( 'VisualEditor' );

Run the update script which will automatically create the necessary database tables that this extension needs.
Follow additional CirrusSearch instructions (i.e. installing composer-based dependencies for Cirrus and Elastic).
If using MediaWiki-Docker, please see MediaWiki-Docker/Extension/CirrusSearch and MediaWiki-Docker/Configuration recipes/ElasticSearch.
Most of the above require no configuration and work out-of-the-box (troubleshooting: Elastica, PageViewInfo, MobileFrontend, VisualEditor)
Done – Navigate to Special:Version on your wiki to verify that the extension is successfully installed.

Vagrant installation:

If using Vagrant , install with vagrant roles enable growthexperiments addlink --provision

Other extensions (optional)

PageImages — Used in link recommendation tasks (for showing images from the corresponding link recommendation articles).
Instrumentation — For more information, see Wikitech
WikiEditor — The default source editor.
Campaigns — Used to track entry points for account creation.
WikimediaEvents — Used for some of the editing-related instrumentation.
GuidedTour — Used to show tooltips (currently on desktop only).
CLDR — Used for localizing language names.

Enabling Growth features

Go to Special:Preferences (while logged in)
Under User profile section: check Display newcomer homepage, Default to newcomer homepage from username link and Enable the editor help panel

Alternatively, run the following in the browser console:

new mw.Api().saveOptions({
	'growthexperiments-help-panel-tog-help-panel': '1',
	'growthexperiments-homepage-enable': '1',
	'growthexperiments-homepage-pt-link': '1',
});

location.reload()

Mentor dashboard

To enable the mentor dashboard follow the next steps:

Log in as an admin
Go to Special:EditGrowthConfig and set "Minimum number of days a user must be registered to sign up as a mentor" and "Minimum number of edits a user must have made (on any namespace) to sign up as a mentor" to zeros. This will allow anyone to enroll as a mentor.
Go to Special:EnrollAsMentor and submit the form. This will enroll your account as a mentor.
Mentor dashboard should be available at http://localhost:8080/wiki/Special:MentorDashboard
Optional (if you want Mentee overview to display some data): Claim some accounts as your mentees and make some edits with them (note those accounts must have Growth features enabled, see #Enabling Growth features). After that, run the php extensions/GrowthExperiments/maintenance/updateMenteeData.php script.

Suggested Edits

There are three basic approaches for setting up suggested edits, the main functionality of GrowthExperiments, in a developer setup:

Have the extension use the search API of a remote (production) wiki. You won't have to deal with setting up search locally, and you will get a wide range of realistic task suggestions, but the articles suggested for those tasks won't exist on your wiki so most editing-related functionality won't work. This is the easiest way to get the homepage and guidance working in general, and to QA task suggestions. It doesn't work for structured link recommendations though since those aren't enabled on any production wiki yet.
Set up search locally, copy-paste or import a couple articles from some real wiki. This is quite a bit more effort but doable (on Vagrant it should mostly work out of the box). It's good for backend development, probably not really worth the effort for frontend development.
Mock all the backend logic involved with static PHP code. This is nice for frontend work as you get direct and full control over the responses from the backend.

Use remote search

use GrowthExperiments\AqsEditInfoService;
use MediaWiki\MediaWikiServices;

# Use the test config for en.wikipedia.org
$wgGENewcomerTasksConfigTitle = 'mw:Growth/Personalized_first_day/Newcomer_tasks/Prototype/templates/en.json';

# Search for tasks on en.wikipedia.org
$wgGENewcomerTasksRemoteApiUrl = 'https://en.wikipedia.org/w/api.php';

# Get extra data for the task cards from en.wikipedia.org
$wgGERestbaseUrl = 'https://en.wikipedia.org/api/rest_v1';

Set up search locally

In Vagrant search can be set up with vagrant roles enable cirrussearch --provision (but it will be set up automatically as a dependency if you provision the growthexperiments role). For Docker, see MediaWiki-Docker/Extension/CirrusSearch and MediaWiki-Docker/Configuration recipes/ElasticSearch. For other setups, a basic configuration is:

wfLoadExtension( 'CirrusSearch' );
require_once "$IP/extensions/CirrusSearch/tests/jenkins/FullyFeaturedConfig.php";
$wgCirrusSearchServers = [ 'elasticsearch' ];
$wgCirrusSearchWMFExtraFeatures = [
	'weighted_tags' => [ 'build' => true, 'use' => true ]
];

See the setup notes of Extension:CirrusSearch for more details.

You can use the CirrusSearch maintenance script UpdateWeightedTags.php for setting the ORES topics of wiki pages.

To use settings from cswiki:

Follow the ElasticSearch instructions from MediaWiki-Docker/Extension/CirrusSearch and MediaWiki-Docker/Configuration_recipes/ElasticSearch.
Copy the contents of MediaWiki:NewcomerTopicsOres.json to http://localhost:8080/w/index.php?title=MediaWiki:NewcomerTopicsOres.json&action=edit. (must be signed in as an admin on MediaWiki)
Copy the contents of MediaWiki:NewcomerTasks.json to http://localhost:8080/w/index.php?title=MediaWiki:NewcomerTasks.json&action=edit. (must be signed in as an admin on MediaWiki)
Add $wgGENewcomerTasksTopicType = 'ores' to LocalSettings.php.
Add $wgGERestbaseUrl = 'https://cs.wikipedia.org/api/rest_v1'; to LocalSettings.php.

To use settings from other wikis, replace "cs" with the language code of the desired wiki (for MediaWiki:NewcomerTasks.json and $wgGERestbaseUrl).

Mock the backend

use GrowthExperiments\NewcomerTasks\AddImage\SubpageImageRecommendationProvider;
use GrowthExperiments\NewcomerTasks\AddLink\SubpageLinkRecommendationProvider;
use GrowthExperiments\NewcomerTasks\ConfigurationLoader\StaticConfigurationLoader;
use GrowthExperiments\NewcomerTasks\TaskSuggester\StaticTaskSuggesterFactory;
use GrowthExperiments\NewcomerTasks\TaskSuggester\TaskSuggesterFactory;
use GrowthExperiments\NewcomerTasks\TaskType\LinkRecommendationTaskType;
use GrowthExperiments\NewcomerTasks\Task\Task;
use GrowthExperiments\NewcomerTasks\TaskType\TaskType;
use MediaWiki\MediaWikiServices;

# Enable under-development features still behind feature flag:
$wgGENewcomerTasksLinkRecommendationsEnabled = true;
$wgGELinkRecommendationsFrontendEnabled = true;

$wgHooks['MediaWikiServices'][] = function ( MediaWikiServices $services ) {
	$linkRecommendationTaskType = new LinkRecommendationTaskType( 'link-recommendation', TaskType::DIFFICULTY_EASY, [] );

	# Mock the configuration, which would normally be at MediaWiki:NewcomerTaskConfig.json, to have just one 'link-recommendation' task type.
	$services->redefineService( 'GrowthExperimentsNewcomerTasksConfigurationLoader', function ( MediaWikiServices $services ) use ( $linkRecommendationTaskType ) {
		return new StaticConfigurationLoader( [ $linkRecommendationTaskType ] );
	} );

	# Mock the task suggester to specify what article(s) will be suggested.
	$services->redefineService( 'GrowthExperimentsTaskSuggesterFactory', function ( MediaWikiServices $services ) use ( $linkRecommendationTaskType ): TaskSuggesterFactory {
		return new StaticTaskSuggesterFactory( [
			new Task( $linkRecommendationTaskType, new TitleValue( NS_MAIN, 'Douglas Adams' ) ),
		] );
	} );
};

# Set up SubpageLinkRecommendationProvider, which will take the recommendation from the article's /addlink.json subpage,
# e.g. [[Douglas Adams/addlink.json]]. The output of https://addlink-simple.toolforge.org can be copied there.
$wgHooks['MediaWikiServices'][] = SubpageLinkRecommendationProvider::class . '::onMediaWikiServices';
$wgHooks['ContentHandlerDefaultModelFor'][] = SubpageLinkRecommendationProvider::class . '::onContentHandlerDefaultModelFor';
# Same for image recommendations, with addimage.json and http://image-suggestion-api.wmcloud.org/?doc
$wgHooks['MediaWikiServices'][] = SubpageImageRecommendationProvider::class . '::onMediaWikiServices';
$wgHooks['ContentHandlerDefaultModelFor'][] = SubpageImageRecommendationProvider::class . '::onContentHandlerDefaultModelFor';

Link recommendations

Your local development environment can query the "external traffic" production release of the link recommendation service. Access is proxied via the api-gateway (api.wikimedia.org) which requires an access token to utilize the API. That means you need to generate a personal access token (https://api.wikimedia.org/wiki/Authentication#Personal_API_tokens), when you will use the access token as the value for GELinkRecommendationServiceAccessToken in LocalSettings.php:

$wgGELinkRecommendationServiceWikiIdMasquerade = 'cswiki';
$wgGELinkRecommendationServiceAccessToken = "#access token from api.wikimedia.org";
$wgGELinkRecommendationServiceUrl = 'https://api.wikimedia.org/service/linkrecommendation';

// Optionally set the fallback on DB miss variable. In practice, if you are
// using refreshLinkRecommendations.php and importOresTopics.php maintenance
// scripts locally, then you probably want this config flag set to false (default)
// $wgGELinkRecommendationFallbackOnDBMiss = true;

To allow non-existent pages (red links) as suggested edits, add the following to LocalSettings.php

$wgGEDeveloperSetup = true;

To update the link recommendations for each topic:

php /path/to/mediawiki/extensions/GrowthExperiments/maintenance/refreshLinkRecommendations.php

To see which articles have corresponding link recommendation tasks:

Go to your local wiki
Search for hasrecommendation:link

Image recommendations

API: 2022 production edition

Register an owner-only OAuth app (with default settings) at https://meta.wikimedia.org/wiki/Special:OAuthConsumerRegistration/propose/oauth2 then set the following values in LocalSettings.php:

$wgGEImageRecommendationApiHandler = "actionapi";
$wgGEImageRecommendationServiceWikiIdMasquerade = "cswiki";
$wgGEImageRecommendationServiceUrl = 'https://cs.wikipedia.org/w/api.php';
$wgGEImageRecommendationServiceAccessToken = '...';
$wgGEImageRecommendationServiceUseTitles = true;

API: MVP edition

To use the old proof-of-concept image suggestions API with the Wikipedia matching the wiki's language, set

$wgGEImageRecommendationServiceUrl = 'https://image-suggestion-api.wmcloud.org';
$wgGEImageRecommendationApiHandler = 'mvp';

Other settings

To use Commons as a file repository (without this the image metadata won't work), set

$wgUseInstantCommons = true;

To tag an article for image recommendation task:

php /path/to/mediawiki/extensions/CirrusSearch/maintenance/UpdateWeightedTags.php --tagType 'recommendation.image' --page $articleTitle
php /path/to/mediawiki/maintenance/runJobs.php

To see which articles have corresponding image recommendation tasks:

Go to your local wiki
Search for hasrecommendation:image

Seeding Articles from Czech Wikipedia

Make sure that http://localhost:8080/wiki/MediaWiki:NewcomerTasks.json is the same as that in Czech Wikipedia.
Get article titles for Growth tasks from Czech Wikipedia using the special page ApiSandBox with the following parameters:
```
#action=query&format=json&export=1&exportnowrap=1&exportschema=0.11&generator=growthtasks
```
This link provides an example. Then copy the response content to a local file, eg: tasks.xml
Import the XML dump by running the following command inside mediawiki installation directory
```
php maintenance/run.php importDump --report=10 tasks.xml
```
Create "Module:Wikidata" on your local site. Copy/paste the Lua source of Module:Wikidata at cs.wikipedia ([1]) and save it.
Update secondary tables (to get accurate information on Special:RecentChanges and Special:Statistics)
```
php maintenance/run.php rebuildrecentchanges
php maintenance/run.php initSiteStats
```

Update ElasticSearch

php maintenance/run.php update --quick
php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php && \
php extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip && \
php extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse && \
php maintenance/run.php runJobs

User impact

(this is work in progress and will change)

By default, user impact functionality is configured to use a subpage provider, ie. for every user impact data will be read from User:<username>/userimpact.json, in the format used by [Expensive]UserImpact::jsonSerialize().

To use real data, you can do something like

use GrowthExperiments\UserImpact\UserImpactLookup;
use MediaWiki\MediaWikiServices;

$wgHooks['MediaWikiServices'][] = function ( MediaWikiServices $services ) {
    $services->redefineService( 'GrowthExperimentsUserImpactLookup',
        function ( MediaWikiServices $services ): UserImpactLookup {
            return $services->get( '_GrowthExperimentsUserImpactLookup_Computed' );
        }
    );
} );

Seeding ORES topics

To import ORES topics for articles imported from a production wiki (English Wikipedia in the example below), run

php extensions/GrowthExperiments/maintenance/importOresTopics.php --apiUrl https://en.wikipedia.org/w/api.php --wikiId enwiki --count 20 --verbose

To set ORES topics for some article manually, use

php extensions/CirrusSearch/maintenance/UpdateWeightedTags.php --tagType classification.ores.articletopic --tagName 'Geography.Regions.Americas.Central America' --weight 750 --tagName 'Media.Media*' --weight 800 --page={yourPageTitle}

Depending on how the wiki is configured, you might need to manually execute the jobs the above commands scheduled:

php maintenance/runJobs.php --type cirrusSearchElasticaWrite