Lazy loading of references on Russian Wikipedia

On the 1st September, lazy loading references was disabled on Russian Wikipedia after both images and references were enabled back in July. The beneficial Lazy loading of images continued to be enabled. Whereas previously all references views were routed via the API, now references would be served in the HTML.

The impact gives the impression that very few users need references in a page view. In a week period, after ending the experiment, an additional 338GB were shipped to Russian Wikipedia and there was only a 0.11GB decrease in bytes shipped via by the API.

In the worse case disabling the experiment increased page load by 2 seconds and first paint by 0.5 seconds.

Impact on performance
[ToDo: Normalise sample size]

Fully load time, first paint and first interactive time were inspected before and after the experiment was disabled.

First paint

DomInteractive

Impact on bytes shipped
The following SQL query was made on all page views for the Russian mobile site: We also had to consider the increased load on the API to retrieve references. The bytes shipped by the API before and after the change to the references api were considered using the following query:

Questions and answers
Q: Can we attribute this change to lazy loaded references?

A: There were no known changes that rolled out during this period that we'd expect to cause such a large increase in bytes shipped by the HTML.

Similar to how we did the analysis for lazy loading images, it's quite possible the Russian Wikipedia project did a lot of editing that week to reduce the size of pages, but that would be a lot of editing for such a large amount of bytes!

One theory might be that during the experiment traffic contributed to this increase in bytes shipped, but looking at the graph you can see this is not the case.

A 338 gb additional shipped page HTML on mobile is quite significant and had that impacted all traffic on desktop as well that would have been noticed by ops, so it's safe to say that there were no core changes that may have caused this! We could analyse desktop traffic for the same period if we lack that confidence, but I feel it would be a lot of effort for little gain.