Extension talk:MobileFrontend/A/B testing audit

These notes were written for the "Audit/document mobile team's bucketing & A/B testing infrastructure" spike.

How are we bucketing users?
We bucket logged in users based on their ID. There are two buckets, Control and TestA, so the bucketing is done by a simple mod 2 split. This bucket is being logged as data for the MobileWebEditing schema.

It's worth noting that all users are assigned a token when local storage is available (http://caniuse.com/#search=WebStorage).

How does this differ from Growth's A/B testing infrastructure?
For experiment on logged in users only, the GettingStarted extension used the same bucketing mechanism.

When experimenting on anonymous users, all users were assigned a token (generated with, though using   was discussed) and that token was logged as data in all events. The token was used to bucket the users in much the same way as above and that bucket was also logged as data in all events. This was done so that there was a one piece of code that was responsible for bucketing users, not three (the front- and back-ends and the researcher doing the experimental analysis).

The token was stored in a cookie for (ideally) 90 days. This was done so that, most importantly, the value was available on the server-side but there were also concerns that the Resource Loader was filling up local storage with cached modules.

Do we need to make any changes to our A/B testing infrastructure in order to run our first WikGrok A/B test
No. There's already enough in place to bucket the user.