Community metrics

How is the MediaWiki / Wikimedia tech community doing? Let's analyze the data available in order to highlight the contributors and areas setting an example, and also the bottlenecks or inactive corners requiring our attention.

Your feedback and requests are welcome in Phabricator (project Analytics-Tech-community-metrics). You can also comment at the discussion page and at the Analytics mailing list.

Median age of open changesets
"Time from last patchset" in days

Open changesets waiting for review
"Waiting for review"

New changesets submitted per month
The "submitted" line

Active Gerrit code review users per month
{ "legends": [{"stroke": "color","title": "type","fill": "color"}], "scales": [ {     "type": "time", "name": "x", "domain": {"data": "chart","field": "x"}, "zero": false, "range": "width", "nice": true },   {      "clamp": true, "type": "linear", "name": "y", "domain": {"data": "chart","field": "y"}, "zero": true, "range": "height", "nice": true },   {      "domain": {"data": "chart","field": "series"}, "type": "ordinal", "name": "color", "range": "category10" } ],  "version": 2, "marks": [ {     "type": "group", "marks": [ {         "properties": { "enter": { "y": {"scale": "y","field": "y"}, "x": {"scale": "x","field": "x"}, "stroke": {"scale": "color","field": "series"}, "strokeWidth": {"value": 2.5} },           "update": {"stroke": {"scale": "color","field": "series"}}, "hover": {"stroke": {"value": "red"}} },         "type": "line" }     ],      "from": { "data": "chart", "transform": [{"groupby": ["series"],"type": "facet"}] }   }  ],  "height": 350, "axes": [{"scale": "x","type": "x"},{"scale": "y","type": "y"}], "data": [ {     "name": "chart", "format": {"type": "json","parse": {"x": "date"}}, "values": [ {"y": 205,"series": "uploaders","x": "Sep 2014"}, {"y": 202,"series": "uploaders","x": "Oct 2014"}, {"y": 188,"series": "uploaders","x": "Nov 2014"}, {"y": 209,"series": "uploaders","x": "Dec 2014"}, {"y": 207,"series": "uploaders","x": "Jan 2015"}, {"y": 210,"series": "uploaders","x": "Feb 2015"}, {"y": 218,"series": "uploaders","x": "Mar 2015"}, {"y": 202,"series": "uploaders","x": "Apr 2015"}, {"y": 207,"series": "uploaders","x": "May 2015"}, {"y": 209,"series": "uploaders","x": "Jun 2015"}, {"y": 194,"series": "uploaders","x": "Jul 2015"}, {"y": 193,"series": "uploaders","x": "Aug 2015"}, {"y": 213,"series": "uploaders","x": "Sep 2015"}, {"y": 213,"series": "uploaders","x": "Oct 2015"}, {"y": 201,"series": "uploaders","x": "Nov 2015"}, {"y": 213,"series": "uploaders","x": "Dec 2015"}, {"y": 228,"series": "uploaders","x": "Jan 2016"}, {"y": 207,"series": "uploaders","x": "Feb 2016"}, {"y": 228,"series": "uploaders","x": "Mar 2016"}, {"y": 211,"series": "uploaders","x": "Apr 2016"}, {"y": 202,"series": "uploaders","x": "May 2016"}, {"y": 204,"series": "uploaders","x": "Jun 2016"}, {"y": 184,"series": "uploaders","x": "Jul 2016"}, {"y": 196,"series": "uploaders","x": "Aug 2016"}, {"y": 215,"series": "uploaders","x": "Sep 2016"}, {"y": 206,"series": "uploaders","x": "Oct 2016"}, {"y": 210,"series": "uploaders","x": "Nov 2016"}, {"y": 226,"series": "uploaders","x": "Dec 2016"}, {"y": 181,"series": "reviewers","x": "Sep 2014"}, {"y": 176,"series": "reviewers","x": "Oct 2014"}, {"y": 170,"series": "reviewers","x": "Nov 2014"}, {"y": 170,"series": "reviewers","x": "Dec 2014"}, {"y": 187,"series": "reviewers","x": "Jan 2015"}, {"y": 190,"series": "reviewers","x": "Feb 2015"}, {"y": 184,"series": "reviewers","x": "Mar 2015"}, {"y": 183,"series": "reviewers","x": "Apr 2015"}, {"y": 178,"series": "reviewers","x": "May 2015"}, {"y": 188,"series": "reviewers","x": "Jun 2015"}, {"y": 200,"series": "reviewers","x": "Jul 2015"}, {"y": 189,"series": "reviewers","x": "Aug 2015"}, {"y": 195,"series": "reviewers","x": "Sep 2015"}, {"y": 192,"series": "reviewers","x": "Oct 2015"}, {"y": 180,"series": "reviewers","x": "Nov 2015"}, {"y": 181,"series": "reviewers","x": "Dec 2015"}, {"y": 198,"series": "reviewers","x": "Jan 2016"}, {"y": 180,"series": "reviewers","x": "Feb 2016"}, {"y": 181,"series": "reviewers","x": "Mar 2016"}, {"y": 182,"series": "reviewers","x": "Apr 2016"}, {"y": 179,"series": "reviewers","x": "May 2016"}, {"y": 165,"series": "reviewers","x": "Jun 2016"}, {"y": 163,"series": "reviewers","x": "Jul 2016"}, {"y": 169,"series": "reviewers","x": "Aug 2016"}, {"y": 181,"series": "reviewers","x": "Sep 2016"}, {"y": 169,"series": "reviewers","x": "Oct 2016"}, {"y": 186,"series": "reviewers","x": "Nov 2016"}, {"y": 181,"series": "reviewers","x": "Dec 2016"}, {"y": 113,"series": "committers","x": "Sep 2014"}, {"y": 123,"series": "committers","x": "Oct 2014"}, {"y": 119,"series": "committers","x": "Nov 2014"}, {"y": 112,"series": "committers","x": "Dec 2014"}, {"y": 117,"series": "committers","x": "Jan 2015"}, {"y": 125,"series": "committers","x": "Feb 2015"}, {"y": 121,"series": "committers","x": "Mar 2015"}, {"y": 117,"series": "committers","x": "Apr 2015"}, {"y": 119,"series": "committers","x": "May 2015"}, {"y": 132,"series": "committers","x": "Jun 2015"}, {"y": 132,"series": "committers","x": "Jul 2015"}, {"y": 126,"series": "committers","x": "Aug 2015"}, {"y": 130,"series": "committers","x": "Sep 2015"}, {"y": 131,"series": "committers","x": "Oct 2015"}, {"y": 125,"series": "committers","x": "Nov 2015"}, {"y": 119,"series": "committers","x": "Dec 2015"}, {"y": 125,"series": "committers","x": "Jan 2016"}, {"y": 126,"series": "committers","x": "Feb 2016"}, {"y": 128,"series": "committers","x": "Mar 2016"}, {"y": 127,"series": "committers","x": "Apr 2016"}, {"y": 125,"series": "committers","x": "May 2016"}, {"y": 121,"series": "committers","x": "Jun 2016"}, {"y": 114,"series": "committers","x": "Jul 2016"}, {"y": 129,"series": "committers","x": "Aug 2016"}, {"y": 132,"series": "committers","x": "Sep 2016"}, {"y": 126,"series": "committers","x": "Oct 2016"}, {"y": 134,"series": "committers","x": "Nov 2016"}, {"y": 130,"series": "committers","x": "Dec 2016"} ]   }  ],  "width": 800 }

MediaWiki Core code review
Number of MediaWiki Core changesets waiting for review in Gerrit (CR0 or CR+1):

Age in days of open MediaWiki Core patchsets:

Active users in Phabricator
Monthly active users in Bugzilla (from 2013-02 to 2014-10) and in Phabricator (from 2014-12 to last month).

New accounts in Phabricator
New Phabricator accounts created every month. About 4003 users have registered to Wikimedia Phabricator between its creation on September 2014 and October 2015.

wikimedia.biterg.io
wikimedia.biterg.io is the Wikimedia Tech community metrics dashboard. It was preceded by korma.wmflabs.org until 2016. Data sources include Git and Gerrit repositories, Phabricator's Maniphest (though only basic support as per February 2017), mediawiki.org, some mailing lists, and some IRC channels (Phabricator's Differential will be supported in the future). Its data is refreshed regularly.

Bugs and feature requests about wikimedia.biterg.io can be reported in Wikimedia Phabricator's Analytics-Tech-community-metrics project.

wikimedia.biterg.io offers:
 * Drill down: clicking an element and a filtered view will be applied
 * Time frame selection
 * Sharing URLs for certain views or embedding views
 * Exporting data
 * API access via the ElastiSearch API
 * Wikimedia administrators to create widget and panels themselves
 * an advanced filter search box

User interface


The top bar lists Dashboards (also called Panels). By default the  is chosen. Each dashboard offers numerous widgets, and a result list at the bottom of the page (commits in Git, emails in mailing lists, etc.).

The interactive Widgets at the bottom display the actual data. Some panels support clicking displayed items to get more specific information about those items and some panels also allow downloading and exporting the displayed data as CSV or JSON.

You can share URLs of dashboards with applied filters by selecting the Share icon to the right of the Advanced filter field.

Applying filters
In the right corner of the top bar, the Time filter allows adjusting the time span of all the data being displayed in the widgets.

Some widgets allow creating Filters: The mouse pointer turns into a plus symbol when hovering over a listed panel item and clicking the item will apply an additional filter for that item. When creating a filter in Kibana, the filter is displayed in green below the Advanced filter text field and is applied to the view. In the screenshot above, 'Bots' and 'Empty commits' are excluded from the data displayed in the panels. When hovering over a filter, you can enable/disable, pin/unpin (the filter will still be applied when you open that page again), invert (e.g. to get all companies listed except for one), remove or edit (e.g. to change the organization name) the filter. The "Actions" menu to the right of the filters offers the same actions to apply them to all filters at once. For more information, see Discover Filters.

The Advanced filter text field allows searching for text in any items (commit messages, user names, repository names, etc.). It allows querying a subset of results provided by the time filter and filters already applied. By default, any free text items in any database columns are included (entering this also resets a search). You basically enter a field name and its value, such as. The query syntax is based on the Lucene query syntax. Also see Kibana Queries and Filters for more information.

As of February 2017 the list of available fields (database columns) is only available to administrators and no auto-complete suggestions are offered (this problem could be worked around by creating a panel that lists the names of the available columns, via "Discover"). To perform advanced search queries, you need to know the names of the available fields.

Some more notes on advanced filters:
 * The type of field (string, number, date, etc.) influences the query syntax
 * Queries are case sensitive
 * You can only create queries which use fields within the respective index (simplified, "indices" in ElasticSearch are kind of databases) that is used in a panel, otherwise the search will return "No results found".
 * Fields not available in an index by default use  for numbers and   for strings

Configuration
Files in https://github.com/Bitergia/mediawiki-repositories define which mailing lists and IRC channels get indexed. It also defines which Git and Gerrit repositories get blacklisted/ignored (for example when they are only imported from upstream sources) in order to not have activity shown that did not happen in our community. Note: As per February 2017 blacklisting is not yet working properly.

It is based on Kibana dashboards and Eleasticsearch. The database provides indexes whose fields are used in panels, widgets and for searches.

Architecture and source code
Details on the underlying software architecture can be found on grimoirelab.github.io. A comprehensive GrimoireLab Training Tutorial is available.

Source code is available. Most code is written in Python. The existing repositories are:
 * : Data retrieval platform which creates JSON files.  contains the available backends. Data is stored in Elasticsearch.
 * : Commander tool to run perceval and set up the panels.
 * : Visualization on top of ElasticSearch. A fork of Kibana which contains changes until they get merged in the upstream code base.
 * : Numerous JSON files. Contains all of the panels currently available for the current architecture.
 * : An incubator for new ideas.
 * : Command line interface to manage the data in our database. For admins, a complete database dump is available as a JSON file which allows manual account merging, updating affiliations, adding country information or marking an account as a bot. See its upstream documentation for more details.

The steps performed are basically: Sources → Data gathering (mining via Perceval) → Data enrichment (e.g. producing indexes in ElasticSearch via GrimoireELK) → Visualization (ElasticSearch and Kibana).

For administrators
The Edit mode for administrators shows more icons on the right of the Advanced filter search field: You can change names of widgets or add a new visualization.

Analyzing specific data
Under Discover, choose a database from the dropdown in the left panel. Then expand the time span. Results are displayed as a list of dropdown data items. Opening a dropdown displays all fields and their values as JSON or a table. A Kibana/ES visualization based on the JSON data is displayed on top. Specific fields can be added as columns to the displayed results by adding/removing those fields in the left panel. It is basically a huge matrix, and if we wanted more data, more fields could be added in the future (e.g. "Gender").

Creating visualizations and dashboards
This section is work in progress.

Further links

 * GrimoireLab training tutorial
 * Extensive user documentation by the Xen Project
 * Kibana User Guide (upstream documentation)
 * Kibana User Guide: Dashboards and Panels (upstream documentation)
 * Building visualizations, GrimoireCon 2017, Brussels
 * Example dashboards of other organizations: Document Foundation, Eclipse, Opnfv, CoreOS, Mozilla's Rust, Linux.

If you would like to see specific customizations, please file a request in Wikimedia Phabricator including a user story.

Other data sources and tools
Git
 * Wikimedia stats in OpenHub/Ohloh including many projects.
 * "How many unique contributors submitted unique pull requests to a https://github.com/wikimedia/ repo" - Python script by marktraceur.

Gerrit
 * Gerrit/Navigation
 * MediaWiki Gerrit stats  (Is it working? 2013-06-28)  and how to query Gerrit data.
 * Number of gerrit committers (marktraceur's bash script)
 * cmd-query for Gerrit.

Phabricator
 * "Phabricator monthly statistics" emails on the wikitech-l mailing list - see its archives.

mediawiki.org
 * monthly Statistics of page views and how the data is gathered.

Mailman
 * Wikimedia Mail Stats: PowerPosters.

Team
Quim Gil and Andre Klapper from the Wikimedia Engineering Community team are coordinating the Metrics Dashboard project, which is being implemented by Bitergia as contractors.

The Bitergia team working in the MediaWiki dashboard is formed by Daniel Izquierdo, Luis Cañas and Jesus Gonzalez Barahona and Alvaro del Castillo as project manager.

The ownership of this project might get transfered to the Wikimedia Analytics team at some point.