Product Analytics/Reporting Guidelines
This page is currently a draft.
Types of reportsEdit
The Product Analytics team produces several types of reports:
- One-time substantial reports after the completion of a project and/or analysis. These are typically published as either a wiki-page or a PDF.
- One shot analysis of specific questions from product teams. These are often published as comments on a Phabricator ticket.
- Recurring reports such as weekly or monthly statistics about a project. These are typically published internally, or available externally through a shared analytics resource.
- All types of reports should describe the purpose of the project and the analysis.
- Metrics should be defined, and reference relevant standardizations where applicable (e.g. standardized retention metric).
- There's no need to create a standalone report if simply reporting the results in a relevant Phab ticket will suffice.
- Provide sufficient detail about the methods used to gather data (e.g. provide a SQL query, define date ranges).
- Define any relevant assumptions.
- Uploading graphs directly to Phabricator is fine. If community members ask, the graphs might also be uploaded to Commons.
- [To be added: something about easy ways to generate tables from data to copy & paste, both from R and Python]
- Recurring reports are expected to be more lightweight and do not need to provide deeper discussions/analysis of the data, they are instead expected to be more of a data summary.
- If the report is generated from a Jupyter Notebook, include a button to show/hide the code (the wmfdata package has a function for this).
- If the report covers a large range of data, consider adding the ability to filter/focus on parts of the data through dynamic graphs. [FIXME: we need to know how to do this]
- Make it clear when the report was last updated and what range of data it contains.
- The report should be easily accessible to relevant stakeholders (e.g. by having it hosted publicly if possible).
Substantial reports (for lack of a better name)Edit
- These reports should include an executive summary of the results and the recommendations that follow from the analysis.
- If the definition of a given method/metric becomes substantial, consider moving it to an Appendix and referring to it as reading for those who are interested.
- Publishing the reports on-wiki (e.g. as a sub-page of a team's pages on MediaWiki-wiki) enables translation into other languages through standardized translation practices on wikis.
- Using the Wikimedia engineering project information template to describe the report and defining the project's start and end dates will automatically categorize the report into the relevant date-based categories for WMF projects. [FIXME: have a Product Analytics-specific template for this]
- For PDF reports, we recommend you use Mikhail's template for this. [FIXME: add a link to the template]