Product Analytics/Reporting Guidelines

From mediawiki.org
Jump to navigation Jump to search

Types of reports[edit]

The Product Analytics team produces several types of reports:

  1. One-time substantial reports after the completion of a project and/or analysis. These are typically published as either a wiki-page or a PDF.
  2. One shot analysis of specific questions from product teams. These are often published as comments on a Phabricator ticket.
  3. Recurring reports such as weekly or monthly statistics about a project. These are typically published internally, or available externally through a shared analytics resource.

Ideas[edit]

  • All types of reports should describe the purpose of the project and the analysis.
  • Metrics should be defined, and reference relevant standardizations where applicable (e.g. standardized retention metric).
  • There's no need to create a standalone report if simply reporting the results in a relevant Phab ticket will suffice.

Phab tickets[edit]

  • Provide sufficient detail about the methods used to gather data (e.g. provide a SQL query, define date ranges).
  • Define any relevant assumptions.
  • Uploading graphs directly to Phabricator is fine. If community members ask, the graphs might also be uploaded to Commons.
  • [To be added: something about easy ways to generate tables from data to copy & paste, both from R and Python]

Recurring reports[edit]

  • Recurring reports are expected to be more lightweight and do not need to provide deeper discussions/analysis of the data, they are instead expected to be more of a data summary.
  • If the report is generated from a Jupyter Notebook, include a button to show/hide the code (the wmfdata package has a function for this).
  • If the report covers a large range of data, consider adding the ability to filter/focus on parts of the data through dynamic graphs. [FIXME: we need to know how to do this]
  • Make it clear when the report was last updated and what range of data it contains.
  • The report should be easily accessible to relevant stakeholders (e.g. by having it hosted publicly if possible).

Substantial reports (for lack of a better name)[edit]

  • These reports should include an executive summary of the results and the recommendations that follow from the analysis.
  • If the definition of a given method/metric becomes substantial, consider moving it to an Appendix and referring to it as reading for those who are interested.
  • Publishing the reports on-wiki (e.g. as a sub-page of a team's pages on MediaWiki-wiki) enables translation into other languages through standardized translation practices on wikis.
  • Using the Wikimedia engineering project information template to describe the report and defining the project's start and end dates will automatically categorize the report into the relevant date-based categories for WMF projects. [FIXME: have a Product Analytics-specific template for this]
  • For PDF reports, we recommend you use Mikhail's template for this. [FIXME: add a link to the template]