Extension:MachineVision/Developers

This page documents the internals of the MachineVision extension, with a focus on the backend (PHP) logic.

Overview
When a new image is uploaded to Wikimedia Commons, the MachineVision extension triggers a delayed job request to ensure that the image is still present (i.e., not deleted), and if so, request and store image label suggestions generated by one or more machine vision labeling providers. These label suggestions are then filtered and served to reviewers on the Special:SuggestedTags page on Commons. Accepted label suggestions are saved to the image's structured data as depicts (P180) statements.

When label suggestions are received for an image, an Echo event is fired to notify the uploader that image labels suggestions are available for review, according to the uploader's notification preferences.

The extension is designed to support arbitrary machine vision providers (including issuing requests to multiple providers simultaneously), but the only provider for which support is currently implemented is Google Cloud Vision.

Image
Images are stored by their SHA1 hash in the  table. This means that if an image file is uploaded that is identical to one for which a record exists in the DB, it is the same image for the MachineVision extension's purposes, and labels will not be requested again.

The extension only handles bitmap images and disregards all other file types.

Label
A label is stored as a Wikidata item ID (Q-number) in the  table. Human-readable labels associated with the item ID are fetched from Wikidata at the point of presentation to the end-user. A label will be associated with an image no more than once, even if the label is subsequently suggested by a different machine vision provider.

Suggestion
A suggestion (stored in the  table) refers to a single instance of a label being suggested for an image. There may be more than one suggestion that refers to an image-label pair, that is, one for each provider that suggests the same label for a given image.

As of September 2020, since there has only ever been one provider configured (namely Google Cloud Vision), there should be a one-to-one relationship between labels and suggestions in practice.

Waiting period
A waiting period is enforced between upload time and the submission of an image to a machine vision provider for label suggestions. This is to reduce the likelihood of making a labeling request for an image that is soon to be deleted. As of September 2020, the waiting period is 48 hours. This value is configured in

Review state
Review state is a critical concept in the MachineVision extension, because it governs which images are presented on Special:SuggestedTags, and to which audiences. It is important to note that, in the extension's internal logic, review states apply to labels rather than to images (on which see "Data model" under "Quirks and gotchas" below). The review states are represented as integers, with a default state of 0 (unreviewed). Possible states include the following:


 * Unreviewed (0): The default review state. The label may be presented in either the "popular" or "personal uploads" tab on Special:SuggestedTags.

Data storage
The extension tables (see Extension:MachineVision/Schema) are hosted in production in the  (x1) cluster and   database.

Label suggestion lifecycle
TODO

Developer setup
TODO

Image and label filtering
Label suggestions have multiple filters applied in  before storage, and each operates differently.

The first filtering pass, based on, is intended to withhold images completely from being shown on Special:SuggestedTags. If a label in  is among the suggested labels returned for an image, the initial review state for all suggested labels is set to WITHHOLD_ALL, which has the effect of excluding it completely. The image is not shown in either the "popular" or "personal uploads" tab on Special:SuggestedTags. The labels are, however, retained in the database.

The second filtering pass, based on, conditionally withholds images from the "popular" tab. If an image receives a SafeSearch rating that exceeds the allowed value on any of the configured dimensions, it is withheld from the "popular" tab but still available to the uploader in the "personal uploads" tab on Special:SuggestedTags. All suggested labels are retained in the database.

The third and final pass, based on, is intended to discard specific label suggestions judged not to be useful to the projects. Suggestions corresponding to labels in  are simply discarded before the remaining suggested labels are stored.

Redirects, deletions, & concept mapping maintenance
TODO

Data model
TODO