Reading/Web/Quantitative Testing

The following plan will be the default option for Quantitative testing for the Reading Web Team on all upcoming features. Details will be determined on a per-feature basis. Development of a feature should not begin prior to communicating a testing plan.

Reading Web Quantitative Testing Requirements

 * 1) Identify relevant metrics [PM, DA, DSGN]
 * 2) List questions to be answered by data (e.g.: does new feature / UI change improve reader retention? Does it attract enough usage?), perhaps drawing from earlier generative research
 * 3) Maybe: do retrospective analysis​ (observational analysis) of existing data from status quo
 * 4) Maybe: coordinate with qualitative testing
 * 5) Decide on use of exploratory data analysis vs. A/B testing
 * 6) Define metrics
 * 7) Maybe: Define success criteria and/or practical significance levels
 * 8) Add quantitative testing tasks to EPIC of the feature
 * 9) Build out feature, schema, and testing structure (potentially A/B test, but not necessarily) Note: Work on the feature must not begin prior to establishing metrics.
 * 10) Spec instrumentation, create/update schema page on Meta, privacy considerations, define purging strategy [DA, engineers, PM]
 * 11) Seek review from Analytics Engineering/Research about privacy considerations and purging strategy [DA, PM]
 * 12) Define sampling method, sampling rates, test venues (wikis) and duration [DA, PM, eng]
 * 13) Implement instrumentation [eng]
 * 14) Test instrumentation [eng, DA] - DA and engineers must meet and review prior to deploying new instrumentation
 * 15) Maybe: Update cookie documentation if needed [eng]
 * 16) Notify communities about test [CL, PM]
 * 17) Roll out feature to beta mode (unless stated otherwise)
 * 18) Roll out feature to 1-2 wikis for a pre-determined test
 * 19) Launch test - default scope to the experimental sample (if not defined otherwise)
 * 20) Monitor live data, sanity checks [DA]
 * 21) Deactivate test [eng]
 * 22) Run test for X months (default to 1 month)
 * 23) Analyze results after first X weeks (default to 2 weeks), publish results, communicate
 * 24) Graduate feature
 * 25) Cake