Growth/Growth 2014/Deployment checklist

General checklist that can be applied to any and all experiments, especially while reviewing on labs/testwiki/etc.

Deployment steps
If we didn't do one of these, there should be a good reason:


 * 1) Deployment to Piramido (our labs instance)
 * 2) * ensure piramido running the wmf branch matching enwiki
 * 3) * ensure all relevant extensions on piramido are updated to master or deployment branch
 * 4) Verify minimum testing checklist.
 * 5) Deployment to test and/or test2 (test2 is recommended)
 * 6) * We get code running on testwiki "for free" early during deployment when we update machine fenari but message resources often remain garbled until we complete deployment.
 * 7) * How can we deploy revised code only to test2?
 * I think this requires two changes to InitialiseSettings.php
 * 1) Verify data collection, re-verify minimum testing checklist
 * 2) All systems go: Deploy to production
 * 3) If there are i18n changes, be sure to use scap or the alternative.
 * 4) Verify correct production experience
 * 5) Announce as necessary to the community. Typically this is on the technical Village Pump on the appropriate project.

Testing checklist

 * Minimum for all products:
 * 1) Does experiment visually comply with design reqs
 * 2) Does experiment comply with functional reqs
 * 3) Perform cross browser check (manual)
 * At least spot check that the page fully loads in all the named B grade browsers, and a few As (IE 9, latest Opera). These are selected since they may not be used day-to-day by engineers.  It's nice if you can test behavior, but there may be time constraints.
 * 1) Data verification - is all data being stored in the right place and in the right way? Do events collected match the schema? (may be verified by engineering/analysts)


 * For A/B or multivariate tests:
 * 1) Verify that opt-out is respected
 * 2) Verify bucketed experience - users in experiment receive correct treatment
 * 3) Verify bucketed experience - users in control group receive correct treatment
 * 4) Verify non-bucket experience - users not in the experiment experience standard WP experience
 * Additional things which are smart to test:
 * 1) Test articles with special elements - semi protected, coordinates, featured articles, infoboxes
 * 2) Test articles with special characters perform appropriately (incl. in title)
 * 3) Test redirected articles perform appropriately
 * 4) Test disambiguation pages perform appropriately
 * 5) Test with major gadgets turned on - no conflicts. esp. applies to Twinkle and Navigation Popups
 * 6) Test performance on long articles and/or other long pages if applicable
 * 7) Test in other popular skins, esp. Monobook
 * 8) Test correct limiting of experience by namespace (depending on reqs)