User:Dan Shick (WMDE)/Drafts/Doodorial
Contents
- Part 1
- Get started with the mwcli development environment
- Get a development version of MediaWiki running locally
- Part 2
- Load an extension
- Make a first extension
- Part 3
- Look at general extension mechanisms in MediaWiki
- Add some basic first extension functionality
- Part 4
- Look at making some real world changes to real world extensions
- Part 5:
- Install the Wikibase extension and get the extension working you your local development environment
- Part 6:
- Take a look at the current Wikidata / Wikibase stable Interface policy and the MediaWiki stable interface policy
- An overview of the current PHP hooks Wikibase provides
- Part 7:
- Walkthroughs of some example extensions
- Part 8:
- Mob programming, build a bulk entity creation API.
Getting set up
- Linux, Mac or Windows with WSL operating system, running on AMD64/x86
- Docker installed and able to run hello world example
Docker images
While you are waiting please get a head start by downloading these docker images
docker pull defreitas/dns-proxy-server:2.19.0docker pull jwilder/nginx-proxy:0.10docker pull docker-registry.wikimedia.org/dev/stretch-php74-fpm:3.0.0docker pull docker-registry.wikimedia.org/dev/buster-apache2:2.0.0docker pull mariadb:10.8 |
IDE
*
- Auto completion
- Code cleanup
- Easy navigation and documentation displays
- etc.
-
- Free 30 day trial https://www.jetbrains.com/phpstorm/
Install mwcli
mwcli contains a Docker-based development environment for MediaWiki.
This is the development environment that we will be using.
We need to install this using the following guide:
Follow the guide linked below:
https://www.mediawiki.org/wiki/Cli/guide/Installation
Mw version
Using mwcli to create a development environment
Once we have mwcli installed, we can set up the development environment for the first time.
This will also:*
- install needed dependencies
- install an initial development site
Follow the guide linked below:
https://www.mediawiki.org/wiki/Cli/guide/Docker-Development-Environment/First-Setup
Once doing this you should have a local MediaWiki install running in the development environment running at default.mediawiki.mwdd.localhost using the port that you chose during setup (default 8080)
More on mwcli
There are many parts of the development environment that we are using that we have not yet used.
You should find the help test of the cli helpful for discovering these features.
- mw docker stop: Stops all currently running containers
- mw docker start: Starts containers that were running before
- mw docker destroy: Deletes all containers and associated data (such as databases)
- mw docker mailhog create: Creates a mailhog service for receiving email
- mw docker phpmyadmin create: Creates a phpmyadmin service for viewing sql databases
You can find the full online reference for the CLI at https://www.mediawiki.org/wiki/Cli/ref/mw_docker
Environment check
Let’s make sure that our development environment is still running.
mw docker docker-compose ps |
You should see that most services show a “State” of “Up”, with one or two showing an “Exit 0”.
If they instead show as existed, then please restart the services.
mw docker resume |
If the list of services is empty then you must have run `destroy` at some point and will need to reinitialize your development environment.
mw docker mediawiki create
mw docker mysql create
mw docker mediawiki install —dbtype=mysql
|
You should then be able to see your wiki
http://default.mediawiki.mwdd.localhost:8080/ << You may need to change the port to be your own
You can find this with
mw docker env get PORT |
Writing some code
Load the BoilerPlate extension
We now have MediaWiki code on your machine in the directory that you selected earlier.
You’ll need to open your IDE so that you can see this code.
In VSCode it’ll look something like this:
Now let’s add the BoilerPlate extension, a mostly blank extension template that displays 'Hello, world!' like text on pages to serve as an example.
https://www.mediawiki.org/wiki/Extension:BoilerPlate
Clicking “browse repository” in the side bar will take you to the git repo
https://gerrit.wikimedia.org/g/mediawiki/extensions/BoilerPlate
We can then clone the repository into the mediawiki extensions directory.
cd $(mw docker mediawiki where)/extensionsgit clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/BoilerPlate" |
The result of this should look something like the below screenshot:
And finally we can enable the extension within MediaWiki by altering our LocalSettings.php file to add the following line:
wfLoadExtension( 'BoilerPlate' ); |
At this point your LocalSettings.php should look something like this.
And if you navigate to the Special:Version page of MediaWiki you should see that the Boiler plate extension is loaded.
Success!
You are running MediaWiki in a development environment and have loaded an additional extension!
Create your own extension
General Guide: https://www.mediawiki.org/wiki/Manual:Developing_extensions
We are going to make a custom extension called “TutorialWorld”
Extracted from the guide, to start you will need to:
- Make a “TutorialWorld” directory in the extensions directory
- Create an extension.json file with some basic information https://www.mediawiki.org/wiki/Manual:Developing_extensions#Registering_features_with_MediaWiki
- Set manifest_version to 2 and a type of “other”
You should end up with something that looks like this:
{"name": "TutorialWorld","author": "Adam","url": "https://www.mediawiki.org/wiki/Extension:TuitorialWorld","description": "Magic things for TutorialWorld","version": "0.1","license-name": "GPL-2.0-or-later","type": "other","manifest_version": 2} |
wfLoadExtension( 'TutorialWorld' ); |
If you navigate back to Special:Version, you should now see your extension in the installed extensions list.
Inside BoilerPlate
The extension.json file for BoilerPlate specifies what the extension is, how it works, and how it connects to MediaWiki.
Full details of the individual parts can be found at https://www.mediawiki.org/wiki/Manual:Extension.json/Schema
In the case of BoilerPlate this is…
This stuff appears on Special:Version among other places
Adding the BeforePageDisplay hook
One of the features that the BoilerPlate extension contains is the ability to add some text to the bottom of all pages (or “vandalize” them).
This can be enabled by setting the `wgBoilerPlateVandalizeEachPage` setting to true in your LocalSettings.php
$wgBoilerPlateVandalizeEachPage = true; |
We can see this setting defined in the extension.json file above.
Once set, if you reload the MainPage, you should see the extension adding text to the end of the page.
At this stage it will be beneficial to open your browser developer console, and disable caching for requests. Particularly if you don’t see this text appear right away.
We can see the code that makes this happen in Hooks.php
This is a namespaced class called Hooks
The full name including namespace is MediaWiki\Extension\BoilerPlate\Hooks
The class implements a Hook interface from MediaWiki called BeforePageDisplayHook.
This interface defines an interface covering the one function implemented in the class called onBeforePageDisplay which contains the functional code.
The code that runs when the hook is called:#
- If it is true
- Add an oojs-ui-core module (defined by MediaWiki)
- Add some HTML to the bottom of the page
If we jump back to extension.json we can see how this function connects to MediaWiki and the hook system.
- A HookHandler is registered with the name “BoilerPlateHooks” pointing at the Hooks class that we just looked at above.
- The specific hook “BeforePageDisplay” is registered pointing to the hook handler “BoilerPlateHooks”
So from MediaWiki to Extension the flow looks something like this:
MediaWiki code -> Named hook -> Hook Handler -> Hooks class -> Hook interface method -> Extension code
Now that we have looked at an example of the BeforePageDisplay hook being used.
Let’s try and implement some different text being added to every page.
We will dedicate ~ 15 minutes to individually working through this problem and asking questions.
After 15 minutes we will go through the bullet point lists with a screen share to fill in any gaps.
Throughout these instructions, try to use the BoilerPlate extension files for inspiration.
- Create an src directory in the TutorialWorld extension
- Register the AutoloadNamespaces for the extension in extension.json
- Create a new class implementing the BeforePageDisplayHook interface in the src directory (Use different text!)
- Register the class as a HookHandler in extension.json
- Register the Hook in extension.json
As a stretch goal, try to make the text configurable!
Your resulting page might look something like this (Note; the “goat”):
Alternate ways to extend MediaWiki
Hooks are one of the major ways to extend MediaWiki or other extension functionality.
There are many hooks in MediaWiki https://www.mediawiki.org/wiki/Manual:Hooks#Available_hooks
And many more hooks in extensions…
Other common extension methods would be:*
- Special pages https://www.mediawiki.org/wiki/Manual:Special_pages
All of these methods follow the same rough layout as hooks:*
- Register the mechanism in extension.json in the appropriate place
- Implement the appropriate code per the extension mechanism
General notes*
- Docker Engine install doesn’t install docker-compose (so hello world will work, but starting with mwcli will complain)
- Mwcli:other notes section was done together with the segment beforehand, instead of together with the pair programming segment.
- Looking for the right hooks seems to be a skill in itself. Does any documentation for this exist?
Wikibase for local development
We aim to develop some code that extends Wikibase at the end of this workshop.
So let’s install Wikibase into our local development environment.
This will be very similar to when we fetched the BoilerPlate extension in workshop 1.
cd $(mw docker mediawiki where)/extensionsgit clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/Wikibase" |
TODO
Wikibase has composer dependencies that we need to also download making use of the composer merge plugin.
Create a file in the mediawiki/corei directory called `composer.local.json` with the following content.
{ "extra": { "merge-plugin": { "include": [ "extensions/*/composer.json", "skins/*/composer.json" ] } }} |
This example file is included in `composer.local.json-sample`, so you can simply run the following:
cd $(mw docker mediawiki where)
cp composer.local.json-sample composer.local.json
|
We then need to make composer update its dependencies.
mw docker mediawiki composer update |
Once we have the code on our system, we can go ahead and load the extension by altering our LocalSetting.php file to add the following lines.
( for now we will only load the Repository part of Wikibase. )
## Wikibase RepositorywfLoadExtension( 'WikibaseRepository', "$IP/extensions/Wikibase/extension-repo.json" );require_once "$IP/extensions/Wikibase/repo/ExampleSettings.php"; |
Note: Ticket relating to getting rid of the requirement for ExampleSettings.php to be loaded
Once loaded, we also need to run update.php
mw docker mediawiki exec -- php maintenance/update.php --quick |
We should then be able to navigate to Special:Version on our wiki, and see wikibase installed there
You can go ahead and try to create a new item using Special:NewItem to ensure everything is working correctly.
Stable interface policies
There are a couple of stable interface policies that will be relevant to MediaWiki and Wikibase development.
Note: The Wikidata stable interface policy was written primarily with Wikidata.org in mind, and does not mention Wikibase, or Wikibase releases much in its current state.
It would likely make sense to iterate on this…
Key points summarized:
-
- Changes of stable interfaces (such as public APIs) will be communicated
- Things considered stable
- JSON and RDF output
- Wikibase Web API (api.php)
- Linked data interface (Special:EntityData)
- Wikidata Query Service
- Wikibase LUA Library
- JavaScript Hooks documented in hooks-js.md
- A selection of things considered UNSTABLE
- Raw Wikibase content stored by MediaWiki and returned by the MediaWiki core API
- Wikibase PHP code
- Wikibase JavaScript code
- The HTML DOM structure
-
- Using Code
- It is generally stable to call public methods on a class instance.
- It is generally not stable to construct a class (instantiate).
- It is generally not stable to extend a class (subclass) and not stable to implement an interface.
- There is a documented deprecation and removal process
- There is a documented definition of an extension “ecosystem” that is recognized
Wikibase PHP hooks
There is developer documentation for Wikibase PHP hooks
Looking at the current Repository PHP hooks*
- WikibaseRepoDataTypes
- Examples usages:
- WikibaseRepoEntityTypes
- Example usages adding entity types:WikibaseLexeme
- Example usages modifying entities (altering search behaviour):
- Additional EntityTypes also generally specify a default namespace and or slot for their content to exist in
- WikibaseRepoEntityNamespaces
- Examples:
- WikibaseMediaInfo (in a dedicated slot in the file namespace)
- WikibaseLexeme (if enabled, in the configured LexemeNamespace)
- WikibaseChangeNotification
- Example usage:
- WikibaseQualityConstraints (Schedules jobs after entity edits)
- WikibaseContentLanguages
- Example usage:
- GetEntityContentModelForTitle
- Example usage:
- WikibaseRepoOnParserOutputUpdaterConstruction
- Allows extensions to register extra EntityParserOutputUpdater implementations.
- Example usage:
- GetEntityByLinkedTitleLookup
- Allows extensions to add custom EntityByLinkedTitleLookup services.
- Example usage:
- Very old, should probably be removed…
- WikibaseTextForSearchIndex
- Unused
- WikibaseContentModelMapping
Break…
Walkthrough: WikibaseManifest
WikibaseManifest is an extension that combines metadata about a Wikibase installation exposing it as a simple API. The goal is to help toolmakers write tools that can target any Wikibase.
Taking a look at extension.json
The extension makes use of:*
- Code is located in the includes directory and MediaWiki\Extension\WikibaseManifest namespace
- 1 REST API route is registered at /wikibase-manifest/v0/manifest
- MediaWiki i18n / Localization system
- MediaWiki Service registration system
The extension is made up of around 23 actual code files.
Some Wikibase services / configurations are used:*
- In order to report via the API the RDF URIs that are used
-
- In order to report the namespaces that entities exist in locally
You can see example API output at https://addshore-alpha.wiki.opencura.com/w/rest.php/wikibase-manifest/v0/manifest
Walkthrough: WikibaseCirrusSearch
This extension implements ElasticSearch-based search for default Wikibase entities (Items and Properties)
Taking a look at extension.json
The extension makes use of:*
- Code is located in the src directory and Wikibase\Search\Elastic namespace
- 9 hooks are used (1 from Wikibase)
- MediaWiki i18n / Localization system
Taking a look at the 1 Wikibase hook that is used.
The hook overrides some fields of entities that are registered.
The fields that are altered can be found in WikibaseSearch.entitytypes.php
Specifically:
Def::ENTITY_SEARCH_CALLBACK
Def::SEARCH_FIELD_DEFINITIONS
Def::FULLTEXT_SEARCH_CONTEXT
These fields relate to the hook used and are documented in the entity types documentation.
Ultimately this extension replaces the default SearchHelper, with a SearchHelper that will look at elastic search.
Walkthrough: AutomatedValues
Wikibase extension that allows defining rules to automatically set labels or aliases based on Statement values.
Taking a look at extension.json
The extension makes use of:*
- Code is located in the src directory and ProfessionalWiki\\AutomatedValues namespace
- 5 hooks are used
- MediaWiki resource loader system is used
Looking at the hook that enabled the main functionality*
- Get the content that is being saved
- If it is an entity with statements
- Apply the relevant defined rules to the content being saved
Walkthrough: WikibaseEDTF
TODO
Break…
Programming time
We will try to peer / mob program a bulk entity creation REST API.
You are more than welcome to follow through the development process locally if you want to. Alternatively focus on the screen share and participating.
Jakob will drive, and Adam will facilitate the direction with input from the whole group.
Relevant reading
Relevant documentation:*
Relevant Wikibase code points:*
- A storage service that will work for ALL registered entity types
- Retrieval: Wikibase\Repo\WikibaseRepo::getEntityStore
- Service: WikibaseRepo.EntityStore
- Interface: EntityStore
- EntityDeserializer
- A deserializer object (JSON -> PHP object) that will work for ALL registered entity types
- Interface: DispatchableDeserializer & Deserializer
NOTE: This is PHP code, so it is unstable per the current stable interface policy.
Planned user input
The user should be able to submit a list of entities to be created by the API in the BODY of a POST request to a `batchcreate` route.
A body could look something like this:
{ "entities":[ { "type": "item", "labels": { "en": { "language": "en", "value": "hello goat" } } }, { "type": "item", "labels": { "en": { "language": "en", "value": "potato land" } } } ]} |
(Entity format is the same as that which is used via wbeditentity)
And a response could look something like this:
[ { "id": "Q1", "rev": 2 }, { "id": "Q2", "rev": 3 }] |
(In the case of an error, return an error key for that entity instead)
Goals
We have a few ordered goals to structure our approach:*
- Implement some code for the route responding with some hardcoded JSON
- Handle input from the user (echo it back)
- Deserialize a list of entities submitted by the user in the body of the request, responding with how many were received
- Create the entities the user is submitting
And a couple of stretch goals:*
- Limit the number of entities that can be created in a request to 100
- Use a factory for the registration of the handler