Growth/Growth 2014/Deployment checklist

General checklist that can be applied to any and all experiments, especially while reviewing on labs/testwiki/etc.

Deployment steps

If we didn't do one of these, there should be a good reason:

1. Deployment to Piramido (our labs instance)
   • ensure piramido running the wmf branch matching enwiki
   • ensure all relevant extensions on piramido are updated to master or deployment branch
2. Verify minimum testing checklist.
3. Deployment to test and/or test2 (test2 is recommended)
   • We get code running on testwiki "for free" early during deployment when we update machine fenari but message resources often remain garbled until we complete deployment.
   • How can we deploy revised code only to test2?

     I don't know if there's a way to do it for revised code. For a new extension, I think this requires two changes to InitialiseSettings.php

4. Verify data collection, re-verify minimum testing checklist
5. All systems go: Deploy to production
   1. If there are i18n changes, be sure to use scap or the alternative.
6. Verify correct production experience
7. Announce as necessary to the community. Typically this is on the technical Village Pump on the appropriate project.

Testing checklist

Minimum for all products

1. Does experiment visually comply with design reqs
2. Does experiment comply with functional reqs
3. Perform cross browser check (manual)

   At least spot check that the page fully loads in all the named B grade browsers, and a few As (IE 9, latest Opera). These are selected since they may not be used day-to-day by engineers. It's nice if you can test behavior, but there may be time constraints.

4. Data verification - is all data being stored in the right place and in the right way? Do events collected match the schema? (may be verified by engineering/analysts)

For A/B or multivariate tests

1. Verify that opt-out is respected (if available)
2. Verify bucketed experience - users in experiment receive correct treatment
3. Verify bucketed experience - users in control group receive correct treatment
4. Verify non-bucket experience - users not in the experiment experience standard WP experience

Additional things which are smart to test

1. Test articles with special elements - semi protected, coordinates, featured articles, infoboxes
2. Test articles with special characters perform appropriately (incl. in title)
3. Test redirected articles perform appropriately
4. Test disambiguation pages perform appropriately
5. Test with major gadgets turned on - no conflicts. esp. applies to Twinkle and Navigation Popups
6. Test performance on long articles and/or other long pages if applicable
7. Test in other popular skins, esp. Monobook
8. Test correct limiting of experience by namespace (depending on reqs)