User:Dan Shick (WMDE)/Drafts/Doodorial

Contents


 * Part 1
 * Get started with the mwcli development environment
 * Get a development version of MediaWiki running locally
 * Part 2
 * Load an extension
 * Make a first extension
 * Part 3
 * Look at general extension mechanisms in MediaWiki
 * Add some basic first extension functionality
 * Part 4
 * Look at making some real world changes to real world extensions
 * Part 5:
 * Install the Wikibase extension and get the extension working you your local development environment
 * Part 6:
 * Take a look at the current Wikidata / Wikibase stable Interface policy and the MediaWiki stable interface policy
 * An overview of the current PHP hooks Wikibase provides
 * Part 7:
 * Walkthroughs of some example extensions
 * Part 8:
 * Mob programming, build a bulk entity creation API.

Getting set up


 * Linux, Mac or Windows with WSL operating system, running on AMD64/x86
 * Docker installed and able to run hello world example
 * Instructions for docker
 * Instructions for hello world

Docker images

While you are waiting please get a head start by downloading these docker images

IDE

To write some code, you’ll need an IDE.

[https://www.codecademy.com/article/what-is-an-ide What is an IDE? ]

Why should I use an IDE? * Syntax highlighting
 * Auto completion
 * Code cleanup
 * Easy navigation and documentation displays
 * etc.

Use the IDE of your choice, but we recommend the 2 below linked to their instructions:


 * https://www.mediawiki.org/wiki/Visual_Studio_Code
 * https://www.mediawiki.org/wiki/JetBrains_IDEs
 * Free 30 day trial https://www.jetbrains.com/phpstorm/

Examples in this presentation use VSCode.

Install mwcli

mwcli contains a Docker-based development environment for MediaWiki.

This is the development environment that we will be using.

We need to install this using the following guide:

Follow the guide linked below:

https://www.mediawiki.org/wiki/Cli/guide/Installation

Mw version

Using mwcli to create a development environment

Once we have mwcli installed, we can set up the development environment for the first time.

This will also:* clone MediaWiki
 * install needed dependencies
 * install an initial development site

Follow the guide linked below:

https://www.mediawiki.org/wiki/Cli/guide/Docker-Development-Environment/First-Setup

Once doing this you should have a local MediaWiki install running in the development environment running at default.mediawiki.mwdd.localhost using the port that you chose during setup (default 8080)



More on mwcli

There are many parts of the development environment that we are using that we have not yet used.

You should find the help test of the cli helpful for discovering these features.


 * mw docker stop : Stops all currently running containers
 * mw docker start : Starts containers that were running before
 * mw docker destroy : Deletes all containers and associated data (such as databases)
 * mw docker mailhog create : Creates a mailhog service for receiving email
 * mw docker phpmyadmin create : Creates a phpmyadmin service for viewing sql databases

You can find the full online reference for the CLI at https://www.mediawiki.org/wiki/Cli/ref/mw_docker

Environment check

Let’s make sure that our development environment is still running.

You should see that most services show a “State” of “Up”, with one or two showing an “ Exit 0 ”.

If they instead show as existed, then please restart the services.

If the list of services is empty then you must have run `destroy` at some point and will need to reinitialize your development environment.

You should then be able to see your wiki

http://default.mediawiki.mwdd.localhost:8080/ << You may need to change the port to be your own

You can find this with

Writing some code

Load the BoilerPlate extension

We now have MediaWiki code on your machine in the directory that you selected earlier.

You’ll need to open your IDE so that you can see this code.

In VSCode it’ll look something like this:



Now let’s add the BoilerPlate extension, a mostly blank extension template that displays 'Hello, world!' like text on pages to serve as an example.

https://www.mediawiki.org/wiki/Extension:BoilerPlate

Clicking “browse repository” in the side bar will take you to the git repo

https://gerrit.wikimedia.org/g/mediawiki/extensions/BoilerPlate



We can then clone the repository into the mediawiki extensions directory.

The result of this should look something like the below screenshot:



And finally we can enable the extension within MediaWiki by altering our LocalSettings.php file to add the following line:

At this point your LocalSettings.php should look something like this.



And if you navigate to the Special:Version page of MediaWiki you should see that the Boiler plate extension is loaded.



Success!

You are running MediaWiki in a development environment and have loaded an additional extension!

Create your own extension

General Guide: https://www.mediawiki.org/wiki/Manual:Developing_extensions

We are going to make a custom extension called “TutorialWorld”

Extracted from the guide, to start you will need to:


 * 1) Make a “TutorialWorld” directory in the extensions directory
 * 2) Create an extension.json file with some basic information https://www.mediawiki.org/wiki/Manual:Developing_extensions#Registering_features_with_MediaWiki
 * 3) Set manifest_version to 2 and a type of “other”

You should end up with something that looks like this:

And then load your extension in LocalSetting.php

If you navigate back to Special:Version, you should now see your extension in the installed extensions list.



Inside BoilerPlate

The extension.json file for BoilerPlate specifies what the extension is, how it works, and how it connects to MediaWiki.

Full details of the individual parts can be found at https://www.mediawiki.org/wiki/Manual:Extension.json/Schema

In the case of BoilerPlate this is…

General metadata

This stuff appears on Special:Version among other places



Where the code is that should be loaded

MediaWIki will instruct PHP to load classes from the directory that you specify



Configuration of the extension

In BoilerPlate this exposes 2 feature toggles for the extension



Hook Handlers, and Hook usages

Hooks allow custom code to be executed when some defined event happens

BeforePageDisplay is used in BoilerPlate which: “Allows last minute changes to the output page, e.g. adding of CSS or JavaScript by extensions.”



Internationalisation files

Adds translations of messages used by the extension to the MediaWiki internationalisation system



Frontend resources

Adds UI resourced to the resource system of MediaWiki



A version of the extension registration manifest



Adding the BeforePageDisplay hook

One of the features that the BoilerPlate extension contains is the ability to add some text to the bottom of all pages (or “vandalize” them).

This can be enabled by setting the `wgBoilerPlateVandalizeEachPage` setting to true in your LocalSettings.php

We can see this setting defined in the extension.json file above.

Once set, if you reload the MainPage, you should see the extension adding text to the end of the page.

At this stage it will be beneficial to open your browser developer console, and disable caching for requests. Particularly if you don’t see this text appear right away.



Hooks

We can see the code that makes this happen in Hooks.php

This is a namespaced class called Hooks

The full name including namespace is MediaWiki\Extension\BoilerPlate\Hooks

The class implements a Hook interface from MediaWiki called BeforePageDisplayHook.

This interface defines an interface covering the one function implemented in the class called onBeforePageDisplay which contains the functional code.

The code that runs when the hook is called:# Checks an extension configuration variable
 * 1) If it is true
 * 2) * Add an oojs-ui-core module (defined by MediaWiki)
 * 3) * Add some HTML to the bottom of the page



If we jump back to extension.json we can see how this function connects to MediaWiki and the hook system.


 * 1) A HookHandler is registered with the name “BoilerPlateHooks” pointing at the Hooks class that we just looked at above.
 * 2) The specific hook “BeforePageDisplay” is registered pointing to the hook handler “BoilerPlateHooks”



So from MediaWiki to Extension the flow looks something like this:

MediaWiki code -> Named hook -> Hook Handler -> Hooks class -> Hook interface method -> Extension code

Implementing BeforePageDisplay

Now that we have looked at an example of the BeforePageDisplay hook being used.

Let’s try and implement some different text being added to every page.

We will dedicate ~ 15 minutes to individually working through this problem and asking questions.

After 15 minutes we will go through the bullet point lists with a screen share to fill in any gaps.

Throughout these instructions, try to use the BoilerPlate extension files for inspiration.


 * 1) Create an src directory in the TutorialWorld extension
 * 2) Register the AutoloadNamespaces for the extension in extension.json
 * 3) Create a new class implementing the BeforePageDisplayHook interface in the src directory (Use different text!)
 * 4) Register the class as a HookHandler in extension.json
 * 5) Register the Hook in extension.json

As a stretch goal, try to make the text configurable!

Your resulting page might look something like this (Note; the “goat”):



Alternate ways to extend MediaWiki

Hooks are one of the major ways to extend MediaWiki or other extension functionality.

There are many hooks in MediaWiki https://www.mediawiki.org/wiki/Manual:Hooks#Available_hooks

And many more hooks in extensions…

Other common extension methods would be:* API endpoints
 * Action api https://www.mediawiki.org/wiki/API:Extensions
 * REST api https://www.mediawiki.org/wiki/API:REST_API/Extensions
 * Special pages https://www.mediawiki.org/wiki/Manual:Special_pages

All of these methods follow the same rough layout as hooks:* Read the MediaWiki documentation on the extension mechanism
 * Register the mechanism in extension.json in the appropriate place
 * Implement the appropriate code per the extension mechanism

General notes* Some docker images don't work well for ARM cpus. (for me(dennis) everything works fine : ))
 * Docker Engine install doesn’t install docker-compose (so hello world will work, but starting with mwcli will complain)
 * Mwcli:other notes section was done together with the segment beforehand, instead of together with the pair programming segment.
 * Looking for the right hooks seems to be a skill in itself. Does any documentation for this exist?

Wikibase for local development

We aim to develop some code that extends Wikibase at the end of this workshop.

So let’s install Wikibase into our local development environment.

This will be very similar to when we fetched the BoilerPlate extension in workshop 1.

TODO

git submodule update --init --recursive

Wikibase has composer dependencies that we need to also download making use of the composer merge plugin.

Create a file in the mediawiki/corei directory called `composer.local.json` with the following content.

This example file is included in `composer.local.json-sample`, so you can simply run the following:

We then need to make composer update its dependencies.



Once we have the code on our system, we can go ahead and load the extension by altering our LocalSetting.php file to add the following lines.

( for now we will only load the Repository part of Wikibase. )

Note: Ticket relating to getting rid of the requirement for ExampleSettings.php to be loaded

Once loaded, we also need to run update.php

We should then be able to navigate to Special:Version on our wiki, and see wikibase installed there



You can go ahead and try to create a new item using Special:NewItem to ensure everything is working correctly.

Stable interface policies

There are a couple of stable interface policies that will be relevant to MediaWiki and Wikibase development.

Note: The Wikidata stable interface policy was written primarily with Wikidata.org in mind, and does not mention Wikibase, or Wikibase releases much in its current state.

It would likely make sense to iterate on this…

Key points summarized:


 * Wikidata / Wikibase stable Interface policy
 * Changes of stable interfaces (such as public APIs) will be communicated
 * Things considered stable
 * JSON and RDF output
 * Wikibase Web API (api.php)
 * Linked data interface (Special:EntityData)
 * Wikidata Query Service
 * Wikibase LUA Library
 * JavaScript Hooks documented in hooks-js.md
 * A selection of things considered UNSTABLE
 * Raw Wikibase content stored by MediaWiki and returned by the MediaWiki core API
 * Wikibase PHP code
 * Wikibase JavaScript code
 * The HTML DOM structure
 * MediaWiki stable interface policy
 * Using Code
 * It is generally stable to call public methods on a class instance.
 * It is generally not stable to construct a class (instantiate).
 * It is generally not stable to extend a class (subclass) and not stable to implement an interface.
 * There is a documented deprecation and removal process
 * There is a documented definition of an extension “ecosystem” that is recognized

Wikibase PHP hooks

There is developer documentation for Wikibase PHP hooks

Looking at the current Repository PHP hooks* Addition of DataTypes and EntityTypes
 * WikibaseRepoDataTypes
 * Examples usages:
 * Math
 * Score
 * WikibaseEdtf
 * WikibaseLocalMedia
 * WikibaseRepoEntityTypes
 * Example usages adding entity types:WikibaseLexeme
 * WikibaseMediaInfo
 * Example usages modifying entities (altering search behaviour):
 * WikibaseCirrusSearch
 * WikibaseLexemeCirrusSearch
 * Additional EntityTypes also generally specify a default namespace and or slot for their content to exist in
 * WikibaseRepoEntityNamespaces
 * Examples:
 * WikibaseMediaInfo (in a dedicated slot in the file namespace)
 * WikibaseLexeme (if enabled, in the configured LexemeNamespace)
 * WikibaseChangeNotification
 * Example usage:
 * WikibaseQualityConstraints (Schedules jobs after entity edits)
 * WikibaseContentLanguages
 * Example usage:
 * WikibaseLexeme
 * GetEntityContentModelForTitle
 * Example usage:
 * WikibaseMediaInfo
 * WikibaseRepoOnParserOutputUpdaterConstruction
 * Allows extensions to register extra EntityParserOutputUpdater implementations.
 * Example usage:
 * WikibaseLexeme


 * GetEntityByLinkedTitleLookup
 * Allows extensions to add custom EntityByLinkedTitleLookup services.
 * Example usage:
 * WikibaseMediaInfo
 * Very old, should probably be removed…
 * WikibaseTextForSearchIndex
 * Unused
 * WikibaseContentModelMapping

Break…

Walkthrough: WikibaseManifest

WikibaseManifest is an extension that combines metadata about a Wikibase installation exposing it as a simple API. The goal is to help toolmakers write tools that can target any Wikibase.

Taking a look at extension.json

The extension makes use of:* 3 configuration options
 * Code is located in the includes directory and MediaWiki\Extension\WikibaseManifest namespace
 * 1 REST API route is registered at /wikibase-manifest/v0/manifest
 * MediaWiki i18n / Localization system
 * MediaWiki Service registration system

The extension is made up of around 23 actual code files.

Some Wikibase services / configurations are used:* RdfVocabulary
 * In order to report via the API the RDF URIs that are used
 * LocalEntitySource
 * In order to report the namespaces that entities exist in locally

You can see example API output at https://addshore-alpha.wiki.opencura.com/w/rest.php/wikibase-manifest/v0/manifest

Walkthrough: WikibaseCirrusSearch

This extension implements ElasticSearch-based search for default Wikibase entities (Items and Properties)

Taking a look at extension.json

The extension makes use of:* 12 configuration options
 * Code is located in the src directory and Wikibase\Search\Elastic namespace
 * 9 hooks are used (1 from Wikibase)
 * MediaWiki i18n / Localization system

Taking a look at the 1 Wikibase hook that is used.

The hook overrides some fields of entities that are registered.

The fields that are altered can be found in WikibaseSearch.entitytypes.php

Specifically:

Def::ENTITY_SEARCH_CALLBACK

Def::SEARCH_FIELD_DEFINITIONS

Def::FULLTEXT_SEARCH_CONTEXT

These fields relate to the hook used and are documented in the entity types documentation.

Ultimately this extension replaces the default SearchHelper, with a SearchHelper that will look at elastic search.

Walkthrough: AutomatedValues

Wikibase extension that allows defining rules to automatically set labels or aliases based on Statement values.

Taking a look at extension.json

The extension makes use of:* 3 configuration options
 * Code is located in the src directory and ProfessionalWiki\\AutomatedValues namespace
 * 5 hooks are used
 * MediaWiki resource loader system is used

Looking at the hook that enabled the main functionality* onMultiContentSave
 * Get the content that is being saved
 * If it is an entity with statements
 * Apply the relevant defined rules to the content being saved

Walkthrough: WikibaseEDTF

TODO

Break…

Programming time

We will try to peer / mob program a bulk entity creation REST API.

You are more than welcome to follow through the development process locally if you want to. Alternatively focus on the screen share and participating.

Jakob will drive, and Adam will facilitate the direction with input from the whole group.

Relevant reading

Relevant documentation:* https://www.mediawiki.org/wiki/API:REST_API/Extensions#Defining_routes
 * https://www.mediawiki.org/wiki/API:REST_API/Extensions#Handling_requests
 * https://www.mediawiki.org/wiki/Manual:Extension.json/Schema#AvailableRights
 * https://www.mediawiki.org/wiki/Manual:Extension.json/Schema#GroupPermissions

Relevant Wikibase code points:* EntityStore
 * A storage service that will work for ALL registered entity types
 * Retrieval: Wikibase\Repo\WikibaseRepo::getEntityStore
 * Service: WikibaseRepo.EntityStore
 * Interface: EntityStore
 * EntityDeserializer
 * A deserializer object (JSON -> PHP object) that will work for ALL registered entity types
 * Retrieval: Wikibase\Repo\WikibaseRepo::getAllTypesEntityDeserializer
 * Service: WikibaseRepo.AllTypesEntityDeserializer
 * Interface: DispatchableDeserializer & Deserializer

NOTE: This is PHP code, so it is unstable per the current stable interface policy.

Planned user input

The user should be able to submit a list of entities to be created by the API in the BODY of a POST request to a `batchcreate` route.

A body could look something like this:

(Entity format is the same as that which is used via wbeditentity)

And a response could look something like this:

(In the case of an error, return an error key for that entity instead)

Goals

We have a few ordered goals to structure our approach:* Register a route
 * Implement some code for the route responding with some hardcoded JSON
 * Handle input from the user (echo it back)
 * Deserialize a list of entities submitted by the user in the body of the request, responding with how many were received
 * Create the entities the user is submitting

And a couple of stretch goals:* Require permissions for API usage
 * Limit the number of entities that can be created in a request to 100
 * Use a factory for the registration of the handler