UserMetrics/Guide

About
UserMetrics is the name of a platform developed by the Wikimedia Foundation to measure user activity based on a set of standardized metrics. Using this platform, a set of key metrics can be selected and applied to a cohort of users to measure their overall productivity. The platform is designed for extensibility (creating new metrics, modifying metric parameters) and to support various types of cohort analysis and program evaluation in a user-friendly way. It accepts requests via a RESTful API and returns responses in JSON format.

As of May 2013, the UserMetrics API is used internally at the Wikimedia Foundation by the Editor Engagement, Global Education and Grantmaking programs. The scope of the project is being extended to include external customers, researchers, and community members. If you are interested in using the UserMetrics API, please contact [mailto:usermetrics@wikimedia.org usermetrics@wikimedia.org].

Project home page and API home
The project home page and UserMetricsAPI home are here:

Project home page: http://mediawiki.org/wiki/UserMetrics

The project home page is the main developer hub for the project, hosting updates and resources for developers and end users.

UserMetrics API home page: http://metrics.wikimedia.org

The UserMetrics API home page is the public URL of the API. Access to the API is currently restricted to internal users and early testers. To obtain credentials, please contact us at [mailto:usermetrics@wikimedia.org usermetrics@wikimedia.org].

Additional information: code repository and bug reports
If you would like additional information about the project, please see:

Code repository: https://github.com/wikimedia/user_metrics

Bug reports: Bugs and feature suggestions should be reported via Bugzilla. https://bugzilla.wikimedia.org/buglist.cgi?component=User%20Metrics&product=Analytics

Contact us
To reach us and/or obtain help, please write to: [mailto:usermetrics@wikimedia.org usermetrics@wikimedia.org].

To receive updates about any UserMetrics service or data quality issues, please join the umapi-alerts list.

Rationale
The UserMetrics API grew out of a need to study data collected via user tagging User Tagging, which is used to identify groups of users so that they can be studied collectively--all subjects of an experiment, for example, or all users who created accounts at an outreach event. The API permits us to easily and efficiently generate reports that provide information about how groups of users behave, for example, how quickly a group of users became productive editors, or how likely a group is to remain active over time.

A second important aim of the project is to develop a standardized set of metrics Metrics Standardization that permits everyone in the organization to have the same understanding of what we mean when we say “an editor has been retained” or “an editor is active.” Because the UserMetrics API uses a standardized set of metrics, the reports generated by the system can be used together—either within a project (to compare an experimental group to a control group, for example) or across the organization (to give an overall sense of the productivity of various efforts). The system is designed to be both flexible and extensible, so that existing metrics can be customized as needed, and new metrics can be added over time.

A third aim of the project is to provide an intuitive workflow Workflow that can be used by any internal team to access and analyze the data required to evaluate an initiative. Metrics can be easily retrieved from the UserMetrics API home page, or via a client, that can automatically generate reports based on the metrics of interest.

User Tagging
The UserMetrics API is designed to leverage the information assigned and stored via userTags, which permit us to permanently associate an arbitrary set of metadata (e.g., “subject of e3 experiment”) to a registered user of a specific project (e.g., "enwiki"). Tags are associated with a userId at the time of account creation or at the time a user undergoes a specific treatment or participates in a given initiative, and are stored in a repository where they can be accessed by the UserMetricsAPI and used to generate   cohorts. Once a tag has been assigned, it cannot be removed or changed.

UserTags can represent any number of user attributes. Tags can identify users as experimental subjects, or users who have created accounts in response to either calls to action or outreach events, or users who are part of a specific program (e.g., Global Education). UserTags do not reflect data that is already captured by MediaWiki or any data that conflict with our privacy policy.

For more information about userTagging, please see: http://www.mediawiki.org/wiki/Usertagging.

Metrics Standardization
Standardized metrics can be applied to any Wikimedia project to help evaluate the impact of initiatives in an unambiguous and consistent way. The set of metrics used by the UserMetricsAPI can be used at the project level to measure the success of an experimental treatment or outreach initiative, or on the organizational level to compare the impact of projects across the organization. In each case, the qualities of interest—user retention or user contribution (quality, quantity, type)--are measured consistently and clearly defined so that all users can see what the numbers mean.

For more background information, please see: http://meta.wikimedia.org/wiki/Research:Metrics

Workflow
The UserMetricsAPI is designed to streamline the process of obtaining and analyzing the data needed to evaluate projects and initiatives. Any authorized UserMetrics user can access the API to generate reports, depending on permissions associated with her user account. Broadly, the workflow can be described as follows:


 * 1) 	Define cohorts. Cohorts can be defined by specifying custom lists of usernames/userIds, or by selecting and combining existing userTags. Examples of cohorts: Users in E3 experimental group, students enrolled in a Global Education class, VisualEditor adopters, new users registered on mobile devices.
 * 2) 	Measure the quality, productivity, or retention of these cohorts via a standard set of metrics:
 * 3) *	revert rate: proportion of reverted edits within 24 hours of registration.
 * 4) *	threshold: reached is a user makes 1 edit to the main namespace within 24 hours of registration
 * 5) *	blocks: number of times user blocked within 24 hours of registration
 * 6) *… or other metrics..
 * 7) 	Compare cohorts against each other or against a baseline:

Cohort
A cohort is a set of users sharing one or more property or attribute—the time of account creation, for example, or participation in an outreach event or experimental group.

The UserMetricsAPI generates cohorts based on userTag information. At its most basic, a cohort can be identified by a single userTag (e.g., “e3_experimental_group”). Cohorts can also be generated from a combination of multiple tags (“e3_experimental_group” and “e3_control_group”). Tags are combined using Boolean operators to reflect either the union or intersection of the groups. For more information, see multiple-tag cohorts.

Metric
Metrics are well-defined values or sets of values that can be computed for any user registered in Wikimedia projects, and are typically used in aggregate to compare different user groups (i.e., cohorts) against each other. The metrics computed by the UserMetricsAPI help us understand user activity and behavior--from the quality, quantity and type of user contribution, to how well our editors are retained. For example, we could look at the value of the “bytes_added” metric to see how many bytes of content a student has added to a given wiki in the last week, but if we are interested in evaluating the success of her class, we would more likely look at the number of bytes added by the entire class (i.e., the “enwiki_editing_class” cohort). In this case, the bytes_added metric is used to help determine if the class is successful. We could look at additional metrics to provide a fuller picture: the revert rate of student edits, for example, or the  survival rate of users in the student cohort. We can’t directly measure the class’s “success,” but we can measure a number of more concrete quantities that help us determine it and compare it with other classes or other similar initiatives.

All metrics are standardized and clearly defined so that we can easily understand what their values mean and consistently use the same standards to evaluate the efficacy of programs and initiatives over time. Note that metrics are dependent on the context in which they are measured and therefore only make sense in these contexts. An editor with a high revert rate could be a vandal, or an advanced user removing vandalized text. In the case of our class of new enwiki users, a high revert rate is more likely vandalism.

The value of a metric returned for each user may be defined (e.g., “true” to indicate that a user reached a threshold of 1 edit in her first 24 hours of account activity) or undefined, which would be the case if a user has not been active for a full 24 hours, and we do not yet know if she will reach the threshold or not. Defined values may be of different types: Boolean (a true or false value indicating whether a threshold has been reached, for example), integer (e.g., edit count), or float (e.g., proportion of reverted edits to total edits). The value of a metric may change over time. As a user makes additional edits, for example, the size of his contribution changes and the value of metrics, such as ‘bytes_added’ will change accordingly. However, once the time over which a given metric is defined has elapsed (e.g., the first week after registration), the metric should also return the same value.

The set of metrics supported by the UserMetricsAPI is in no way exhaustive. The system has been designed to be easily extensible, so that new metrics can be added and parameterized in different ways.

The backend
The backend of the UserMetrics API consists of two main pieces: the UserTag repository, which contains information about each userTag and the users to which it has been applied, and the UserMetrics engine, which receives metric request URLs and returns metric data as JSON objects.

The UserTag repository
The UserTag repository contains information about each user tag and the users to which it has been applied. In addition, the repository stores the name of the team using the tag, as well as the name of the cohort owner (the person responsible for maintaining cohort membership and keeping cohort information up-to-date). The repository consists of four MySQL tables (usertags_meta, ut_tags, api_user, and api_group). Because these tables may contain sensitive information, they are not publicly accessible. If you need access to specific information stored in the repository, please e-mail [mailto:usermetrics@wikimedia.org usermetrics@wikimedia.org].

The definition of each usertag, as well as other relevant metadata, is stored in the usertags_meta table:



Note that each tag has a unique Id (e.g., 64, in the above example) and a human-readable name (e3_ob4a). All tags are project-specific (e.g., ‘enwiki’). Currently, each usertag is applied to only one project; in the future, tags may be applied to several projects. In this case, the value of the ‘utm_project’ field would contain the names of all relevant projects. The usertags_meta table also contains a description of each tag and numerical codes representing the owner and the team using the tag. Owner and team names can be looked up in the api_user and api_group tables, respectively. The ‘utm_touched’ column represents the date the tag was most recently applied to a user. This information—the Ids of users associated with each tag—is stored in the ut_tag table. Finally, the ‘utm_enabled’ value indicates whether or not the tag is current. A value of ‘1’ indicates that the tag is relevant and should be included in the UserMetrics API application (where it can be selected from a menu and used to define a cohort). A value of ‘0’ indicates that the tag is no longer relevant, and should be archived. All tags—both current and archived—remain in the usertags_meta table.

The usertags table contains the name of the project (e.g., ‘enwiki’ or ‘arwiki’) and the userId of the users associated with each tag. Tags are identified by a unique numeric identifier referenced in the usertags_meta table; further information about each tag can be referenced in the usertags_meta table.

The api_user and api_group tables identify the groups and individuals using the repository. Each group (e.g., ‘e3’ or ‘mobile’) is assigned a unique numerical identification.

The UserMetrics engine and API
The UserMetrics engine is a Python package that includes several modules to process and handle "jobs" or "requests" and return a response and the associated information. The engine takes care of creating metric request objects and querying the databases (primarily the revision, user, page, and logging MySQL tables). The UserMetrics engine can be run as a standalone package, or with the UserMetrics API, which translates request URLs from authorized users into calls to the engine, creates and manages a job queue to best utilize system resources, and caches the response so that it is available for reuse.



When a request is submitted to the UserMetrics engine via the API, the request is first validated by the API to ensure that the URL is well formed and the parameters valid. The API also looks to the cache to see if the query has been submitted previously. If the query already exists, the API returns the cached response. If the request does not yet exist, the API creates a unique object representing the request and schedules a new job.

All new jobs are handled by the Job Scheduler, which queues jobs on arrival and sends those jobs to a helper process to be run. The Job Scheduler knows how many jobs are currently running and, if the maximum number has been reached, will add the new job to a wait queue. As existing jobs complete, the Job Scheduler selects new jobs from the wait queue in the order in which they were added. The Job Scheduler communicates with the Request Notifications Listener, which maintains the job queue, gets jobs, gets the status of jobs, adds jobs, and flags jobs as complete. When a job is complete, the Response Builder constructs a response for caching, caches the response, and notifies the system that the job is complete.

In addition to the job-handling components, the API consists of four primary modules: Views, Session, RequestMeta, and Data. The Views module handles the Flask URL-routing, view implementation, and subsequent template rendering. The Session module extends the Flask log-in package to create an interface for registering and authenticating users. The RequestMeta (RM) module parses request-URLs and creates from each a uniquely identified object that can be passed among the various components of the UserMetrics engine. The Data module handles reading and writing data to the cache, constructing hashes from the RM objects, constructing URLs from RM objects, and rebuilding JSON HTTP responses from cached responses.

Once the Job Scheduler has determined that a job should run, the relevant RM object is passed to a separate helper process, process_metrics (PM), which constructs and validates a cohort from the given information. The PM takes care of mapping the RM parameters to user metrics objects, specifying the parameters that determine the number of worker threads to be used by the metrics object, and whether or not to log the status of the call. Further processing depends on the request type: raw, aggregate, or time series.

Raw requests are handled by a metric object’s ‘process’ method, which implements the logic necessary to generate numerical results by userId. This method is defined for all UserMetric derived classes and takes a list of userIds as an argument. If metric parameters are not passed to the UserMetrics engine as keyword arguments, the metrics object applies its own default values.

Aggregate requests are handled in much the same way as raw requests, only an ‘aggregator’ wrapper method is used to combine the numerical results in a specified way (e.g., sum, mean, median). Aggregators are defined for each metric using method definitions in the Aggregator module.

Time series requests are handled with the ‘timeseries_method’ module. PM makes a call to this module with the time-series request data. See 'Understanding the different types of requests’  for more information about time-series requests.

All metrics objects use the Query module (QM) to extract data from the backend datastore (currently MySQL, though the system may be extended to use additional data sources, as the query logic is abstracted from the metric classes). The Query module formats and builds MySQL queries, enforces security on MySQL requests, and abstracts away the details of obtaining data from the MediaWiki databases. The Query module provides a single hub through which all queries to sensitive data are run, making it possible to secure the databases more easily.

The API writes the metric data objects to cache, both to a persistent backend store (HDFS) and to memory as an ordered dictionary. The system can be easily extended to use memcached or another caching system. API responses are returned via HTTP as JSON objects.

The UserMetrics API
The UserMetrics API receives request URLs and returns metrics data in formatted JSON. Metrics can be generated for any registered user or predefined group of users (i.e., cohort). Each request URL specifies the name of the cohort of interest as well as a metric handle—the friendly name of the metric to be computed, e.g., ‘threshold’--and, optionally, request parameters.

The type of data returned depends on the nature of the metric of interest as well as the type of request submitted (raw, aggregate, time series). All responses are stored in cache, and can be viewed as soon as the data has been retrieved, or at a later date.

In this section, we will look more closely at how to build a request—how to select a cohort, how to use the available metrics and their parameters, and how to use additional parameters to customize the request—as well as how to understand the response.

Overview of workflow
Whether you are running an experiment and are interested in looking at the behavior of a test and control group, or heading an outreach program and are curious about how effective the program is in retaining productive users, the UserMetrics API can be used gather data that sheds light on how the relevant users are interacting with the Wikimedia site.

If one hundred new users create accounts at an outreach event, for example, and you would like to evaluate how productive those users are, you could use the UserMetric API to retrieve the number of edits made by each individual in the group (or, alternatively, the total number of edits made by all individuals in the group) to give a measure of productivity. If the outreach event is focused on training new users, you could look at how quickly those users became active users (or reached some predefined threshold of activity) as an indication of how effective the training was. If you are interested in examining the quality of the contribution, or have concerns that a group of users is doing vandalism, you could look at the revert rate for the group. Whether your question has to do with the volume of user contribution, the quality of that contribution, the user retention, or the contribution type, user metrics can be used—and customized—to help answer it.

Once you have settled on the metric that will best answer your question about a particular cohort of users, you can customize your request with parameters. Parameters are used to specify the time period of interest (one day, one week, three month, one year, etc.), the start and end dates of a time interval of interest, the namespace and project of interest, or an event count (e.g., the number of edits that represent a threshold of interest). A program administrator usually presets the subset of metrics and default parameters most relevant to the project. Unless you need to (or have been asked to) override the defaults, you don't need to make changes before generating a request.

Parameters are also used to indicate the type of the request itself. By default, the system returns the metric value for each user in the cohort (a raw response). Specifying an aggregator parameter (‘sum’ or ‘average,’ for example) instructs the UserMetrics engine to return an aggregate value for the entire cohort. Similarly, the time series parameters instruct the system to return a series of aggregate values, one for each time slice of the defined interval.

The UserMetrics API validates and processes all valid requests. If the request has been submitted previously, the data is immediately pulled from the cache and returned as JSON objects. If a request is new, the engine will process the job, and return a report once the data has been extracted from the relevant databases. Metric data can then be viewed and used by any authenticated UserMetrics user.

Accessing the API
All users can read about the UserMetrics project and available metrics on the | UserMetrics API home page, which is publically accessible.

Only authenticated users may make requests, view cohorts, or retrieve the response generated by a previous request. Currently, authentication credentials are available to internal Wikimedia Foundation staff, early beta testers, and a number of trusted individuals from chapters and other organizations affiliated with the Foundation.

To obtain credentials, please contact [mailto:usermetrics@wikimedia.org usermetrics@wikimedia.org].

Defining and selecting cohorts
Cohorts, which are constructed from data stored in the UserTag repository, are exposed in the UserMetrics application as selectable items. Available cohorts may be used singly (to identify members of a particular test group, for example) or in combination (to identify members of a test group who also responded to a particular outreach event, for example). We are currently working on an interface for adding new cohorts to the system, and the documentation will be updated to reflect that functionality when it becomes available.

Information about cohorts is stored in the UserTag repository, which contains information about the tags that define cohorts. Each userTag has a unique numerical Id as well as a human-legible name, which is displayed in the UserMetrics interface. Additional information about each tag—a description, as well as the name of the person and team using it, for example—can be found in the usertags_meta table.

Single-tag cohorts
A single-tag cohort, as its name implies, is defined by a single userTag. All currently available single-tag cohorts are included in the “List of cohorts” menu in the UserMetrics application. To use one of these cohorts (e.g., the ‘test2’ cohort), simply select it from the cohort menu. The request URL will then reflect the name of the selected cohort:

http://metrics.wikimedia.org/cohorts/test/

Multiple-tag cohorts
A multiple-tag cohort is based on a combination of two or more userTags. Multiple-tag cohorts are defined in the request string using Boolean operators, which combine single-tag cohorts in the specified way (either ‘union’ or ‘intersection’, though other operators may be supported in the future). Note that each userTag in a multiple-tag cohort must be identified by its unique numerical Id (not the human-legible name displayed in the application).

To create a multiple-tag cohort that is the union of two cohorts (i.e., that includes users who appear in either of the two single-tag cohorts) use the ‘~’ syntax. A multiple-tag cohort derived from the union of ‘newUI’ (userTag Id ‘20’) and ‘revisedNewUI’ (userTag Id ‘29’), for example, would be represented by ‘20~29’ in the request URL:

http://metrics.wikimedia.org/cohorts/20~29/

To create a multiple-tag cohort that is the intersection of two cohorts (i.e., that includes only those users that appear in both cohorts), use the ‘&’ syntax:

http://metrics.wikimedia.org/cohorts/20&29/

Multiple-tag cohorts may consist of any number of individual tags and may contain a combination of ‘&’ and ‘~’ operators. If both operators are used, the intersection operator (‘&’) will take precedence. Currently, there is no support for nesting precedence.

Future functionality will include support for working with dynamic cohorts and adding new cohorts via the UserMetrics interface.

Static (fixed-membership) cohorts vs dynamic cohorts
PLACEHOLDER

Adding a custom cohort
PLACEHOLDER

Understanding different types of requests
The data the UserMetrics API returns depend on the type of request submitted to the system:


 * Raw request: By default, the system returns ‘raw data,’ which consists of the userId of the specified users, as well as the value of the metric for each user.
 * Aggregate request: Requests may also contain an aggregator, such as ‘average’ or ‘sum’. When an aggregator is applied, the system returns the aggregated data (e.g., the average edit rate for an entire cohort, rather than individual values for each user).
 * Time-series request: Aggregated data can be further processed via a time-series request. A time-series request returns data from a specified interval (the last month, for example) sliced by day, week, hour, or whatever unit is most relevant to the analysis. The UserMetrics engine calculates an aggregate metric value for each time slice. The aggregate values can be further customized to reflect either the activity by all registered users during the time slice, or only the activity of those users who registered within each slice.
 * Single-user request: A single-user request returns the value of the specified metric for one user. The user is identified by username (userId will be supported in the future) and project (e.g., ‘enwiki’ or ‘arwiki’). If no project is specified, the system returns results from ‘enwiki’ by default.
 * All-user request: An all-user request returns the value of the specified metric for all users of a project (‘enwiki,’ by default). An all-user request is created with the ‘all’ magic keyword, which specifies that metrics be returned for all project users instead of for a cohort drawn from the UserTag repository.

All requests contain the name of the user or cohort of interest, as well as the name of the metric to process and report. Additional parameters that further qualify the request (e.g., an aggregator or time series) or the metric itself (e.g., the number of edits that define a threshold: 1, 5, 10, etc.) may also be specified. If no additional parameters are specified, the UserMetrics API will use default values, which we look at in more depth in the ‘Available Metrics’ section.

In this section, we will look at the components of each type of request abstractly and also provide a working example for each type of request.

Raw
A raw request returns the value of the specified metric for each user. A raw request consists of a cohort name and the name of the desired metric. Metric-specific parameters may be included as well: $$Rr$$ {cohort, metric(params)}

For example, if a number of users (50, in this example) created accounts at an outreach event, and you’d like to see how many edits each of those users made in their first 24 hours, you would specify the ‘outreach_event’ cohort. If this cohort does not yet exist, or is not yet known to UserMetrics, it will have to be created and/or added to the system first. See “How to define and select a cohort” for more information. The edit count for each user in the cohort is returned by the ‘bytes_added’ metric:

Request components: $$R_r$$ {outreach_event_cohort, bytes_added}

Returned data: { ‘user_ID1’: [bytes_added data] ‘user_ID2’: [bytes_added data] ‘user_ID3’: [bytes_added data] ‘user_ID4’: [bytes_added data] … ‘user_ID50’: [bytes_added data] }

Example of a raw request (using the ‘test2’ cohort, which consists of 3 users):

Request URL: http://metrics.wikimedia.org/cohorts/test2/bytes_added Returned data:  "type": "raw", "data": {"15972203": [ 683, 1133, 908, -225, 5 ], "13234584": [0, 0, 0, 0, 0],   	 	  "15972135": [0, 0, 0, 0, 0]}

The above data is excerpted from the actual JSON response, and reflect the userId (e.g., ‘15972203’) the net bytes added (‘683’), the absolute number of bytes added (‘1133’), positive bytes added (‘908’), negative bytes added (‘-225’), and edit count (‘5’). The metric is measured over the default time period: the first 24 hours after registration for each user. We look at full JSON responses in more depth in the “Understanding the response” section. If you are an authenticated UserMetrics API user, you can view the full JSON response for this request by clicking the request URL.

If you would like to see the total edit count for the group—perhaps to compare that sum to a baseline to determine if the cohort is more or less productive—you would use an aggregator. We will extend this example to use an aggregator in the next section.

Aggregate
An aggregator combines the data associated with individual cohort members in the specified way—returning an aggregate response like an average, for example, or a sum. Aggregators are metric-specific. See ‘Metric-specific parameters and aggregators’ for a list of aggregators currently supported by each metric. If you specify an aggregator that is not supported by a metric, or misspell the aggregator keyword, the UserMetrics API will return a raw response for the request.

To create an aggregate request, supply the cohort name, the name of the metric (and, optionally, any metric-specific parameters), and an aggregator:

$$Ra$$ {cohort, metric(params), aggregator}

If we would like to look at the sum of the bytes_added values returned for the users in the ‘outreach_event_cohort’, we would specify: Request components: $$ Ra$$ {outreach_event_cohort, bytes_added(params), aggregator=sum}

Request return: { ‘bytes_added_sum’: [sum of bytes_added by cohort members] }

The UserMetrics system returns one value, the ‘bytes_added_sum’ for the entire cohort.

Example of an aggregate request (using the ‘test2’ cohort, which consists of 3 users):

Request URL: http://metrics.wikimedia.org/cohorts/test2/bytes_added?aggregator=sum Returned data:  "type": "aggregator", "data": [ 683, 1133, 908, -225, 5 ]

The above data is excerpted from the actual JSON response, and reflect the aggregated data for the entire cohort (not the metric values for each individual user, which are not included in the response). In this example, the UserMetrics API returns the sum of the ‘bytes_added’ metric for all cohort users: net bytes added (‘683’), absolute number of bytes added (‘1133’), positive bytes added (‘908’), negative bytes added (‘-225’), and edit count (‘5’).

We look at full JSON responses in more depth in the “Understanding the response” section. If you are an authenticated UserMetrics API user, you can view the full JSON response for this request by clicking the request URL.