User:Phuedx (WMF)/Bonus Exp

Pageviews by logged-in users are served by our application servers. Pageviews by logged-out users are served by our edge caches.

For logged-in users, therefore, we’re free to modify the content of the response however and whenever we choose based on whatever variables are available to the application servers. For example, we can add or remove interface elements, and we can add or remove JavaScript, CSS, interface icons, and localisation text via ResourceLoader given that the user is viewing the page from England at 9:27 AM BST (UTC+1) using Google Chrome. The response generated will only be served to that user.

For logged-out users, however, we can modify the content of the response as above but there are a lot fewer variables available to the application servers. The response generated for the user viewing the page from England at 9:27 AM BST (UTC+1) using Google Chrome can be served to the user viewing the page in New York at 7:49 PM EDT (UTC-4) using Mozilla Firefox.

If we assume that our caching infrastructure is immutable, then we must limit the features that we can experiment on for logged-out users to those delivered after the response and require the user interact with the page, e.g. we could vary the number of search results shown in the search autocomplete feature or we could vary the treatment of the dialog shown when the user clicks on the “X languages” button. We must also be careful not to draw attention to the feature being varied as well as doing so might bias the results of our experiment, e.g. an interface element appearing on the screen and reflowing nearby content. In cases like this, we should deliver an initial response with a placeholder interface element that is then replaced when a variant is selected.

The final constraint is that experiments on features for logged-out users whose unit of variance is the session ID must be run on a per-wiki basis. This is because session IDs are not shared between domains, i.e. the session ID of the user on en.wikipedia.org is different from that on de.wikipedia.org even if the user visits both domains in the same browser at the same time. If an experiment were running on those wikis concurrently, that user may receive a different variant on each.

= Addendum = It is possible to test per-page variations on logged-out users by using page_random, a persistent, random number generated when the page is first created, as the unit of variance. This approach should be limited to features that are invisible to the user, as if it were used on interface elements, say, it would lead to an inconsistent interface between pageviews for all users. An example of when this approach was used is the Readers Web’s sameAs Test.