Product Analytics/Reporting Guidelines

Types of reports
The Product Analytics team produces several types of reports:


 * 1) One-time substantial reports after the completion of a project and/or analysis. These are typically published as either a wiki-page or a PDF.
 * 2) One shot analysis of specific questions from product teams. These are often published as comments on a Phabricator ticket.
 * 3) Recurring reports such as weekly or monthly statistics about a project. These are typically published internally, or available externally through a shared analytics resource.

Ideas

 * All types of reports should describe the purpose of the project and the analysis.
 * Metrics should be defined, and reference relevant standardizations where applicable (e.g. standardized retention metric).
 * There's no need to create a standalone report if simply reporting the results in a relevant Phab ticket will suffice.

Phab tickets

 * Provide sufficient detail about the methods used to gather data (e.g. provide a SQL query, define date ranges).
 * Define any relevant assumptions.
 * Uploading graphs directly to Phabricator is fine. If community members ask, the graphs might also be uploaded to Commons.
 * [To be added: something about easy ways to generate tables from data to copy & paste, both from R and Python]

Recurring reports

 * Recurring reports are expected to be more lightweight and do not need to provide deeper discussions/analysis of the data, they are instead expected to be more of a data summary.
 * If the report is generated from a Jupyter Notebook, include a button to show/hide the code (the wmfdata package has a function for this).
 * If the report covers a large range of data, consider adding the ability to filter/focus on parts of the data through dynamic graphs. [FIXME: we need to know how to do this]
 * Make it clear when the report was last updated and what range of data it contains.
 * The report should be easily accessible to relevant stakeholders (e.g. by having it hosted publicly if possible).

Substantial reports (for lack of a better name)

 * These reports should include an executive summary of the results and the recommendations that follow from the analysis.
 * If the definition of a given method/metric becomes substantial, consider moving it to an Appendix and referring to it as reading for those who are interested.
 * Publishing the reports on-wiki (e.g. as a sub-page of a team's pages on MediaWiki-wiki) enables translation into other languages through standardized translation practices on wikis.
 * Using the Wikimedia engineering project information template to describe the report and defining the project's start and end dates will automatically categorize the report into the relevant date-based categories for WMF projects. [FIXME: have a Product Analytics-specific template for this]
 * For PDF reports, we recommend you use Mikhail's template for this. [FIXME: add a link to the template]