Reading/Web/Lazy loading of images on Japanese Wikipedia

On Tuesday 5th July lazy loaded images was enabled on Japanese Wikipedia for all users. The immediate impact we noticed is that images served to users on 日本 (Japan) dropped from 1.443 MB to 142 kB.

Bytes saved
The bytes saved varies per article. Articles with more images obviously will be reduced more. We can analyze bytes saved by looking at raw access logs and looking at images with an internal referrer. We can also look at specific pages to gauge impact.

Analysis of weeks worth of data
A database dump of the EventLogging data for Japanese Wikipedia was taken from the period 1st June to 12th July. Data 10 days before and after the change was compared and sample size was not normalized. Given the change was deployed on 5th July at 4:47pm PST the following query was used. select * from NavigationTiming_15485142 where wiki = 'jawiki' and event_mobileMode = 'stable' and event_action ='view' and timestamp > 20160625004700 and timestamp < 20160715004700 This SQL query gave a sample of x rows of data. With the result of the following query, using a Python script the following command was run python navtimingcsv.py jawiki.tsv 20160705004700 10

Analysis of 30 days of data
A database dump of the EventLogging data for Japanese Wikipedia was taken from the period 5th June to 4th August. Data 30 days before and after the change was compared. Given the change was deployed on 5th July at 4:47pm PST the following query was used. select * from NavigationTiming_15485142 where wiki = 'jawiki' and event_mobileMode = 'stable' and event_action ='view' and timestamp > 20160601004700 and timestamp < 20160804004700

This SQL query gave a sample of 1249001 rows of data. With the result of the following query, using a Python script the following command was run python navtimingcsv.py japanwiki.tsv 20160705004700 30

Week against week comparisons and longterm data transfer graph
Rollout of lazy loaded images on ja.m.wikipedia.org suggests a nontrivial speed improvement in page fully loaded time (initial lag excluded) and a significant reduction in image bytes shipped per pageview, leading to lighter weight pages.

Note well that the x-axis in the speed graphs is presented in log2n increments.

To reduce noise, navigation timing events are filtered to ensure key fields are present, with an emphasis on anonymous, non-redirected, plain article pageviews. Logged in pageviews for the NavigationTiming queried window comprised only 0.117% of the pageviews in the set and are not used in the NavigationTiming derived calculations.

To simplify analysis for bytes shipped calculations, eligible source HTML pages are constrained as to a relative path   (on language variant wikis this path would be different, but that's out of scope here) for a given wiki without a colon ":" character in the remainder of the path, with a restriction that responses must be HTTP 200s, in order to avoid overcounting of 30x, 40x, or other such spurious responses. This aids in narrowing down the analysis to requests likely to be plain article pageviews. Image bytes are constrained to those served from upload.wikimedia.org with an eligible Referer for the same restriction as page paths.

Changes introduced to support lazy loaded images required modified (slightly increased) JavaScript/CSS/HTML.

jawiki page load speed
Based on weeks just before and just after the transition to lazy loaded images, pages with lazy loaded images loaded faster at the 10th, 50th (median), and 90th percentiles on both HTTP1 and HTTP2. A basic Kolomogorov-Smirnov statistical significance test suggested that the pre- and post-change cohort speed differences were statistically significant.

The following table is based on anonymous pageviews using the query below.

jawiki data transfer
Based on examination of two 7-day periods (24-30 June 2016 vs 8-14 July 2016),  image bytes per pageview were reduced by about 51.37% . This contributed to a decrease of about 9.18% bytes shipped for the modified JavaScript/CSS/HTML plus images as compared to the baseline JavaScript/CSS/HTML and images.

The following graph shows the general trends of images bytes shipped per in-scope pageview, as well in-scope pageviews, core HTML/CSS/JavaScript bytes, and the sum of core HTML/CSS/JavaScript bytes plus image bytes per pageview.

Queries
The following query was used to derive the lag-excluded load time, roughLoadTimeInitialLagExcluded. Note this query also captured data on tlwiki, although the scope of the analysis on this page is jawiki mobile

The following queries were used to derive image bytes transferred using the constraints described above, as well as to support the longer term graph with trend lines.

The raw data received were as follows:

month	day	_c2	_c3 6	30	https://ja.m.wikipedia.org	124180180469 6	5	https://ja.m.wikipedia.org	135720072139 6	6	https://ja.m.wikipedia.org	108178906611 6	7	https://ja.m.wikipedia.org	111401108523 6	8	https://ja.m.wikipedia.org	106164037819 6	9	https://ja.m.wikipedia.org	105406723448 6	10	https://ja.m.wikipedia.org	104839062684 6	11	https://ja.m.wikipedia.org	141886296285 6	12	https://ja.m.wikipedia.org	170638150504 6	13	https://ja.m.wikipedia.org	146692782536 6	14	https://ja.m.wikipedia.org	134084883702 6	15	https://ja.m.wikipedia.org	133562013207 6	16	https://ja.m.wikipedia.org	142537562128 6	17	https://ja.m.wikipedia.org	133251860232 6	18	https://ja.m.wikipedia.org	151938833556 6	19	https://ja.m.wikipedia.org	167860625982 6	20	https://ja.m.wikipedia.org	136443890103 6	21	https://ja.m.wikipedia.org	136692664905 6	22	https://ja.m.wikipedia.org	147929702365 6	23	https://ja.m.wikipedia.org	139453947620 6	24	https://ja.m.wikipedia.org	156084306064 6	25	https://ja.m.wikipedia.org	168950872999 6	26	https://ja.m.wikipedia.org	171282566058 6	27	https://ja.m.wikipedia.org	135543126283 6	28	https://ja.m.wikipedia.org	136833152633 6	29	https://ja.m.wikipedia.org	135462758117 7	1	https://ja.m.wikipedia.org	124684212453 7	2	https://ja.m.wikipedia.org	164418677635 7	3	https://ja.m.wikipedia.org	161752597537 7	4	https://ja.m.wikipedia.org	149993526937 7	5	https://ja.m.wikipedia.org	134035063705 7	6	https://ja.m.wikipedia.org	103235144867 7	7	https://ja.m.wikipedia.org	67828813390 7	8	https://ja.m.wikipedia.org	66337742258 7	9	https://ja.m.wikipedia.org	80564160978 7	10	https://ja.m.wikipedia.org	85920579820 7	11	https://ja.m.wikipedia.org	68779837487 7	12	https://ja.m.wikipedia.org	67698073684 7	13	https://ja.m.wikipedia.org	69773350604 7	14	https://ja.m.wikipedia.org	67633588058 7	15	https://ja.m.wikipedia.org	67486541269 7	16	https://ja.m.wikipedia.org	78015793312 7	17	https://ja.m.wikipedia.org	86470922402 7	18	https://ja.m.wikipedia.org	86544561954 7	19	https://ja.m.wikipedia.org	66590125244 7	20	https://ja.m.wikipedia.org	64112347900 7	21	https://ja.m.wikipedia.org	66050339575 7	22	https://ja.m.wikipedia.org	63700245787 7	23	https://ja.m.wikipedia.org	72510674705 7	24	https://ja.m.wikipedia.org	82103567189 7	25	https://ja.m.wikipedia.org	66774017408 7	26	https://ja.m.wikipedia.org	67152344637 7	27	https://ja.m.wikipedia.org	65026808541 7	28	https://ja.m.wikipedia.org	62521007017 7	29	https://ja.m.wikipedia.org	62639495316 7	30	https://ja.m.wikipedia.org	70456033383 7	31	https://ja.m.wikipedia.org	79507268276 8	1	https://ja.m.wikipedia.org	67572125043 8	2	https://ja.m.wikipedia.org	66961618961 8	3	https://ja.m.wikipedia.org	67092175391 8	4	https://ja.m.wikipedia.org	65062330557

The following queries were used to derive page and JavaScript/CSS bytes (pre- and post-modification for lazy loading) transferred using the constraints described above, as well as to support the longer term graph with trend lines.

The raw data received were as follows:

month	day	uri_host	_c3 6	5	ja.m.wikipedia.org	584043509098 6	6	ja.m.wikipedia.org	453573407856 6	7	ja.m.wikipedia.org	443560946176 6	8	ja.m.wikipedia.org	445979910759 6	9	ja.m.wikipedia.org	500639200613 6	10	ja.m.wikipedia.org	488962065693 6	11	ja.m.wikipedia.org	547200189342 6	12	ja.m.wikipedia.org	593204311330 6	13	ja.m.wikipedia.org	492918438444 6	14	ja.m.wikipedia.org	447036568383 6	15	ja.m.wikipedia.org	453638629857 6	16	ja.m.wikipedia.org	479841968101 6	17	ja.m.wikipedia.org	517480911709 6	18	ja.m.wikipedia.org	551939360543 6	19	ja.m.wikipedia.org	591493939800 6	20	ja.m.wikipedia.org	458977191045 6	21	ja.m.wikipedia.org	454923880318 6	22	ja.m.wikipedia.org	492636495002 6	23	ja.m.wikipedia.org	456603025665 6	24	ja.m.wikipedia.org	490341384015 6	25	ja.m.wikipedia.org	537929362713 6	26	ja.m.wikipedia.org	545503508434 6	27	ja.m.wikipedia.org	431615776632 6	28	ja.m.wikipedia.org	448780648978 6	29	ja.m.wikipedia.org	451773538098 6	30	ja.m.wikipedia.org	423185112608 7	3	ja.m.wikipedia.org	537992180969 7	4	ja.m.wikipedia.org	448322594045 7	5	ja.m.wikipedia.org	445007695821 7	6	ja.m.wikipedia.org	435472597765 7	7	ja.m.wikipedia.org	430814756332 7	8	ja.m.wikipedia.org	461334758600 7	9	ja.m.wikipedia.org	547280892717 7	10	ja.m.wikipedia.org	600062076779 7	11	ja.m.wikipedia.org	473057269970 7	12	ja.m.wikipedia.org	477262829644 7	13	ja.m.wikipedia.org	471436878948 7	14	ja.m.wikipedia.org	472398332176 7	15	ja.m.wikipedia.org	548724119441 7	16	ja.m.wikipedia.org	616527657391 7	17	ja.m.wikipedia.org	639723642046 7	18	ja.m.wikipedia.org	638694292375 7	19	ja.m.wikipedia.org	481106407202 7	20	ja.m.wikipedia.org	473148500657 7	21	ja.m.wikipedia.org	466261658125 7	22	ja.m.wikipedia.org	465249325390 7	23	ja.m.wikipedia.org	492041513150 7	24	ja.m.wikipedia.org	563837017528 7	25	ja.m.wikipedia.org	445503057133 7	26	ja.m.wikipedia.org	447799358192 7	27	ja.m.wikipedia.org	447035047673 7	28	ja.m.wikipedia.org	432588647022 7	29	ja.m.wikipedia.org	447436227804 7	30	ja.m.wikipedia.org	503914286671 7	31	ja.m.wikipedia.org	553840973015 7	1	ja.m.wikipedia.org	424109984113 7	2	ja.m.wikipedia.org	535223235640 8	4	ja.m.wikipedia.org	463721744365 8	1	ja.m.wikipedia.org	465305749921 8	2	ja.m.wikipedia.org	459082686410 8	3	ja.m.wikipedia.org	468718936637

The following queries were used to derive pageviews using the constraints above, as well as to support the longer term graph with trend lines. Practically all matching records were qualified as pageviews, largely ruling out the possibility of image byte transfer counts with proper Referer values being derived from anything other than qualified pageviews.

The raw data received were as follows. Note that the low count "text/html" and "-" content types, as well as the non-pageview records are not used in the calculations on this page. These were included in this query to gauge the magnitude of their potential effect. It would appear their effect was minimal.

6	19	ja.m.wikipedia.org	text/html; charset=UTF-8	true	21683504 6	28	ja.m.wikipedia.org	application/pdf	false	1 6	17	ja.m.wikipedia.org	-	false	4 6	17	ja.m.wikipedia.org	image/png	false	9 6	21	ja.m.wikipedia.org	text/html; charset=UTF-8	false	1 6	21	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17370675 6	17	ja.m.wikipedia.org	text/css; charset=utf-8	false	6 6	20	ja.m.wikipedia.org	text/html	true	2 6	17	ja.m.wikipedia.org	text/javascript	false	2 6	23	ja.m.wikipedia.org	text/html; charset=UTF-8	false	9 6	23	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17673155 6	25	ja.m.wikipedia.org	text/html; charset=UTF-8	false	6 6	25	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20560909 6	6	ja.m.wikipedia.org	text/html; charset=UTF-8	false	6 6	6	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17427607 6	17	ja.m.wikipedia.org	image/vnd.microsoft.icon	false	7 6	27	ja.m.wikipedia.org	text/html; charset=UTF-8	false	3 6	27	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16981545 6	8	ja.m.wikipedia.org	text/html; charset=UTF-8	false	3 6	8	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17168195 6	26	ja.m.wikipedia.org	text/html	true	2 6	29	ja.m.wikipedia.org	text/html; charset=UTF-8	false	1 6	29	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17880569 6	10	ja.m.wikipedia.org	text/html; charset=UTF-8	false	12 6	10	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17312428 6	12	ja.m.wikipedia.org	text/html; charset=UTF-8	false	2 6	12	ja.m.wikipedia.org	text/html; charset=UTF-8	true	21701144 6	14	ja.m.wikipedia.org	text/html; charset=UTF-8	false	15 6	14	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16960827 6	16	ja.m.wikipedia.org	text/html; charset=UTF-8	false	24 6	16	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17600084 6	17	ja.m.wikipedia.org	application/json; charset=utf-8	false	6 6	18	ja.m.wikipedia.org	text/html; charset=UTF-8	false	11 6	18	ja.m.wikipedia.org	text/html; charset=UTF-8	true	19381891 6	20	ja.m.wikipedia.org	text/html; charset=UTF-8	false	11 6	20	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17291241 6	22	ja.m.wikipedia.org	text/html; charset=UTF-8	false	4 6	22	ja.m.wikipedia.org	text/html; charset=UTF-8	true	18665374 6	30	ja.m.wikipedia.org	text/javascript; charset=utf-8	false	1 6	24	ja.m.wikipedia.org	text/html; charset=UTF-8	false	5 6	24	ja.m.wikipedia.org	text/html; charset=UTF-8	true	18789507 6	5	ja.m.wikipedia.org	text/html; charset=UTF-8	false	8 6	5	ja.m.wikipedia.org	text/html; charset=UTF-8	true	21686129 6	26	ja.m.wikipedia.org	text/html; charset=UTF-8	false	4 6	26	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20798241 6	7	ja.m.wikipedia.org	text/html; charset=UTF-8	false	6 6	7	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17492334 6	17	ja.m.wikipedia.org	text/javascript; charset=UTF-8	false	5 6	22	ja.m.wikipedia.org	text/javascript	false	1 6	28	ja.m.wikipedia.org	text/html; charset=UTF-8	false	5 6	28	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17811344 6	9	ja.m.wikipedia.org	text/html; charset=UTF-8	false	3 6	9	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17059342 6	17	ja.m.wikipedia.org	text/javascript; charset=utf-8	false	27 6	30	ja.m.wikipedia.org	text/html; charset=UTF-8	false	4 6	30	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16896524 6	11	ja.m.wikipedia.org	text/html; charset=UTF-8	false	7 6	11	ja.m.wikipedia.org	text/html; charset=UTF-8	true	19594972 6	13	ja.m.wikipedia.org	text/html; charset=UTF-8	false	6 6	13	ja.m.wikipedia.org	text/html; charset=UTF-8	true	18410111 6	21	ja.m.wikipedia.org	text/javascript; charset=utf-8	false	1 6	30	ja.m.wikipedia.org	text/css; charset=utf-8	false	1 6	15	ja.m.wikipedia.org	text/html; charset=UTF-8	false	1 6	15	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17176324 6	17	ja.m.wikipedia.org	image/svg+xml	false	7 6	17	ja.m.wikipedia.org	text/html; charset=UTF-8	false	5 6	17	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17645960 6	17	ja.m.wikipedia.org	text/css; charset=UTF-8	false	1 6	19	ja.m.wikipedia.org	text/html; charset=UTF-8	false	4 7	6	ja.m.wikipedia.org	text/html	true	3 7	28	ja.m.wikipedia.org	text/html; charset=UTF-8	false	14 7	28	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16285305 7	9	ja.m.wikipedia.org	text/html; charset=UTF-8	false	10 7	9	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20806034 7	30	ja.m.wikipedia.org	text/html; charset=UTF-8	false	2 7	30	ja.m.wikipedia.org	text/html; charset=UTF-8	true	18485500 7	11	ja.m.wikipedia.org	text/html; charset=UTF-8	false	6 7	11	ja.m.wikipedia.org	text/html; charset=UTF-8	true	18072054 7	13	ja.m.wikipedia.org	text/html; charset=UTF-8	false	8 7	13	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17788981 7	3	ja.m.wikipedia.org	application/pdf	false	12 7	12	ja.m.wikipedia.org	text/html	true	1 7	15	ja.m.wikipedia.org	text/html; charset=UTF-8	false	6 7	15	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17055753 7	14	ja.m.wikipedia.org	text/html	true	1 7	17	ja.m.wikipedia.org	text/html; charset=UTF-8	false	11 7	17	ja.m.wikipedia.org	text/html; charset=UTF-8	true	21657938 7	28	ja.m.wikipedia.org	image/vnd.microsoft.icon	false	1 7	7	ja.m.wikipedia.org	application/pdf	false	1 7	25	ja.m.wikipedia.org	text/javascript; charset=utf-8	false	1 7	8	ja.m.wikipedia.org	text/javascript; charset=UTF-8	false	7 7	19	ja.m.wikipedia.org	text/html; charset=UTF-8	false	4 7	19	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17341474 7	28	ja.m.wikipedia.org	application/pdf	false	2 7	8	ja.m.wikipedia.org	text/javascript; charset=utf-8	false	1 7	21	ja.m.wikipedia.org	text/html; charset=UTF-8	false	5 7	21	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17240735 7	2	ja.m.wikipedia.org	text/html; charset=UTF-8	false	2 7	2	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20680721 7	23	ja.m.wikipedia.org	text/html; charset=UTF-8	false	4 7	23	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17927678 7	4	ja.m.wikipedia.org	text/html; charset=UTF-8	false	70 7	4	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17534848 7	25	ja.m.wikipedia.org	text/html; charset=UTF-8	false	5 7	25	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16528308 7	6	ja.m.wikipedia.org	text/html; charset=UTF-8	false	8 7	6	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17320422 7	27	ja.m.wikipedia.org	text/html; charset=UTF-8	false	6 7	27	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16801517 7	8	ja.m.wikipedia.org	text/html; charset=UTF-8	false	11 7	8	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17586299 7	25	ja.m.wikipedia.org	image/png	false	1 7	29	ja.m.wikipedia.org	text/html; charset=UTF-8	false	30 7	29	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16521987 7	4	ja.m.wikipedia.org	text/javascript	false	1 7	25	ja.m.wikipedia.org	text/css; charset=utf-8	false	1 7	10	ja.m.wikipedia.org	text/html; charset=UTF-8	false	14 7	10	ja.m.wikipedia.org	text/html; charset=UTF-8	true	22014268 7	25	ja.m.wikipedia.org	text/javascript	false	1 7	31	ja.m.wikipedia.org	text/html; charset=UTF-8	false	5 7	31	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20046905 7	2	ja.m.wikipedia.org	image/vnd.microsoft.icon	false	1 7	12	ja.m.wikipedia.org	text/html; charset=UTF-8	false	5 7	12	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17971521 7	21	ja.m.wikipedia.org	application/pdf	false	1 7	10	ja.m.wikipedia.org	-	false	1 7	14	ja.m.wikipedia.org	text/html; charset=UTF-8	false	8 7	14	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17191513 7	16	ja.m.wikipedia.org	text/html; charset=UTF-8	false	27 7	16	ja.m.wikipedia.org	text/html; charset=UTF-8	true	19963165 7	18	ja.m.wikipedia.org	text/html; charset=UTF-8	false	8 7	18	ja.m.wikipedia.org	text/html; charset=UTF-8	true	22374981 7	20	ja.m.wikipedia.org	text/html; charset=UTF-8	false	10 7	20	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17164645 7	1	ja.m.wikipedia.org	text/html; charset=UTF-8	false	23 7	1	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16583764 7	10	ja.m.wikipedia.org	application/pdf	false	1 7	22	ja.m.wikipedia.org	text/html; charset=UTF-8	false	9 7	22	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16624345 7	3	ja.m.wikipedia.org	text/html; charset=UTF-8	false	4 7	3	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20609077 7	24	ja.m.wikipedia.org	text/html; charset=UTF-8	false	7 7	24	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20685545 7	5	ja.m.wikipedia.org	text/html; charset=UTF-8	false	109 7	5	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17714516 7	25	ja.m.wikipedia.org	application/json; charset=utf-8	false	1 7	26	ja.m.wikipedia.org	text/html; charset=UTF-8	false	2 7	26	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16840162 7	7	ja.m.wikipedia.org	text/html; charset=UTF-8	false	5 7	7	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16916233 8	1	ja.m.wikipedia.org	text/html; charset=UTF-8	false	4 8	1	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17212992 8	3	ja.m.wikipedia.org	text/html; charset=UTF-8	false	11 8	3	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17545536 8	1	ja.m.wikipedia.org	application/pdf	false	1 8	2	ja.m.wikipedia.org	text/html; charset=UTF-8	false	9 8	2	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17167953 8	4	ja.m.wikipedia.org	text/html; charset=UTF-8	false	5 8	4	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17291588

As with any data spanning time series and the myriad complexities involved with different devices and environments, data are subject to fluctuation. However, the data transfer savings are unambiguous, and the large event sampling pool lends a degree of confidence that pages are actually loading faster.

Statistical significance

The Kolomogorov-Smirnov test suggests the differences in the distributions for lazy loaded compared to non-lazy loaded image based page load times for each of HTTP/1 and HTTP/2 are material and we can reject the null hypothesis.

from scipy.stats import ks_2samp

with open('h1_conventional.txt') as f:	a = [int(i) for i in f.readlines]
 * 1) h1_conventional.txt contains 24-30 June 2016 HTTP/1 convetional loading rough load times

with open('h1_lazy.txt') as f:	b = [int(i) for i in f.readlines]
 * 1) h1_lazy.txt contains 8-14 July 2016 HTTP/2 lazy loading rough load times

len(a) 15517 len(b) 15898

ks_2samp(a,b) (0.037166639427270387, 7.1355334748281838e-10)


 * 1) result is ks statistic, p-value. Null hypothesis rejected.

with open('h2_conventional.txt') as f:	c = [int(i) for i in f.readlines]

with open('h2_lazy.txt') as f:	d = [int(i) for i in f.readlines]

len(c) 53735 len(d) 55437

ks_2samp(c,d) (0.020979426651792699, 7.149451368361895e-11)


 * 1) result is ks statistic, p-value. Null hypothesis rejected.

The data used for the two month line graph for bytes shipped, pageviews, etc. is provided here:

In-scope pageviews month	day	uri_host	content_type	is_pageview	pvs 6	5	ja.m.wikipedia.org	text/html; charset=UTF-8	true	21686129 6	6	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17427607 6	7	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17492334 6	8	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17168195 6	9	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17059342 6	10	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17312428 6	11	ja.m.wikipedia.org	text/html; charset=UTF-8	true	19594972 6	12	ja.m.wikipedia.org	text/html; charset=UTF-8	true	21701144 6	13	ja.m.wikipedia.org	text/html; charset=UTF-8	true	18410111 6	14	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16960827 6	15	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17176324 6	16	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17600084 6	17	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17645960 6	18	ja.m.wikipedia.org	text/html; charset=UTF-8	true	19381891 6	19	ja.m.wikipedia.org	text/html; charset=UTF-8	true	21683504 6	20	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17291241 6	21	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17370675 6	22	ja.m.wikipedia.org	text/html; charset=UTF-8	true	18665374 6	23	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17673155 6	24	ja.m.wikipedia.org	text/html; charset=UTF-8	true	18789507 6	25	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20560909 6	26	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20798241 6	27	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16981545 6	28	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17811344 6	29	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17880569 6	30	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16896524 7	1	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16583764 7	2	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20680721 7	3	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20609077 7	4	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17534848 7	5	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17714516 7	6	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17320422 7	7	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16916233 7	8	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17586299 7	9	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20806034 7	10	ja.m.wikipedia.org	text/html; charset=UTF-8	true	22014268 7	11	ja.m.wikipedia.org	text/html; charset=UTF-8	true	18072054 7	12	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17971521 7	13	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17788981 7	14	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17191513 7	15	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17055753 7	16	ja.m.wikipedia.org	text/html; charset=UTF-8	true	19963165 7	17	ja.m.wikipedia.org	text/html; charset=UTF-8	true	21657938 7	18	ja.m.wikipedia.org	text/html; charset=UTF-8	true	22374981 7	19	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17341474 7	20	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17164645 7	21	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17240735 7	22	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16624345 7	23	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17927678 7	24	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20685545 7	25	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16528308 7	26	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16840162 7	27	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16801517 7	28	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16285305 7	29	ja.m.wikipedia.org	text/html; charset=UTF-8	true	16521987 7	30	ja.m.wikipedia.org	text/html; charset=UTF-8	true	18485500 7	31	ja.m.wikipedia.org	text/html; charset=UTF-8	true	20046905 8	1	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17212992 8	2	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17167953 8	3	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17545536 8	4	ja.m.wikipedia.org	text/html; charset=UTF-8	true	17291588

image bytes month	day	_c2	_c3 6	5	https://ja.m.wikipedia.org	135720072139 6	6	https://ja.m.wikipedia.org	108178906611 6	7	https://ja.m.wikipedia.org	111401108523 6	8	https://ja.m.wikipedia.org	106164037819 6	9	https://ja.m.wikipedia.org	105406723448 6	10	https://ja.m.wikipedia.org	104839062684 6	11	https://ja.m.wikipedia.org	141886296285 6	12	https://ja.m.wikipedia.org	170638150504 6	13	https://ja.m.wikipedia.org	146692782536 6	14	https://ja.m.wikipedia.org	134084883702 6	15	https://ja.m.wikipedia.org	133562013207 6	16	https://ja.m.wikipedia.org	142537562128 6	17	https://ja.m.wikipedia.org	133251860232 6	18	https://ja.m.wikipedia.org	151938833556 6	19	https://ja.m.wikipedia.org	167860625982 6	20	https://ja.m.wikipedia.org	136443890103 6	21	https://ja.m.wikipedia.org	136692664905 6	22	https://ja.m.wikipedia.org	147929702365 6	23	https://ja.m.wikipedia.org	139453947620 6	24	https://ja.m.wikipedia.org	156084306064 6	25	https://ja.m.wikipedia.org	168950872999 6	26	https://ja.m.wikipedia.org	171282566058 6	27	https://ja.m.wikipedia.org	135543126283 6	28	https://ja.m.wikipedia.org	136833152633 6	29	https://ja.m.wikipedia.org	135462758117 6	30	https://ja.m.wikipedia.org	124180180469 7	1	https://ja.m.wikipedia.org	124684212453 7	2	https://ja.m.wikipedia.org	164418677635 7	3	https://ja.m.wikipedia.org	161752597537 7	4	https://ja.m.wikipedia.org	149993526937 7	5	https://ja.m.wikipedia.org	134035063705 7	6	https://ja.m.wikipedia.org	103235144867 7	7	https://ja.m.wikipedia.org	67828813390 7	8	https://ja.m.wikipedia.org	66337742258 7	9	https://ja.m.wikipedia.org	80564160978 7	10	https://ja.m.wikipedia.org	85920579820 7	11	https://ja.m.wikipedia.org	68779837487 7	12	https://ja.m.wikipedia.org	67698073684 7	13	https://ja.m.wikipedia.org	69773350604 7	14	https://ja.m.wikipedia.org	67633588058 7	15	https://ja.m.wikipedia.org	67486541269 7	16	https://ja.m.wikipedia.org	78015793312 7	17	https://ja.m.wikipedia.org	86470922402 7	18	https://ja.m.wikipedia.org	86544561954 7	19	https://ja.m.wikipedia.org	66590125244 7	20	https://ja.m.wikipedia.org	64112347900 7	21	https://ja.m.wikipedia.org	66050339575 7	22	https://ja.m.wikipedia.org	63700245787 7	23	https://ja.m.wikipedia.org	72510674705 7	24	https://ja.m.wikipedia.org	82103567189 7	25	https://ja.m.wikipedia.org	66774017408 7	26	https://ja.m.wikipedia.org	67152344637 7	27	https://ja.m.wikipedia.org	65026808541 7	28	https://ja.m.wikipedia.org	62521007017 7	29	https://ja.m.wikipedia.org	62639495316 7	30	https://ja.m.wikipedia.org	70456033383 7	31	https://ja.m.wikipedia.org	79507268276 8	1	https://ja.m.wikipedia.org	67572125043 8	2	https://ja.m.wikipedia.org	66961618961 8	3	https://ja.m.wikipedia.org	67092175391 8	4	https://ja.m.wikipedia.org	65062330557 page bytes month	day	uri_host	_c3 6	5	ja.m.wikipedia.org	584043509098 6	6	ja.m.wikipedia.org	453573407856 6	7	ja.m.wikipedia.org	443560946176 6	8	ja.m.wikipedia.org	445979910759 6	9	ja.m.wikipedia.org	500639200613 6	10	ja.m.wikipedia.org	488962065693 6	11	ja.m.wikipedia.org	547200189342 6	12	ja.m.wikipedia.org	593204311330 6	13	ja.m.wikipedia.org	492918438444 6	14	ja.m.wikipedia.org	447036568383 6	15	ja.m.wikipedia.org	453638629857 6	16	ja.m.wikipedia.org	479841968101 6	17	ja.m.wikipedia.org	517480911709 6	18	ja.m.wikipedia.org	551939360543 6	19	ja.m.wikipedia.org	591493939800 6	20	ja.m.wikipedia.org	458977191045 6	21	ja.m.wikipedia.org	454923880318 6	22	ja.m.wikipedia.org	492636495002 6	23	ja.m.wikipedia.org	456603025665 6	24	ja.m.wikipedia.org	490341384015 6	25	ja.m.wikipedia.org	537929362713 6	26	ja.m.wikipedia.org	545503508434 6	27	ja.m.wikipedia.org	431615776632 6	28	ja.m.wikipedia.org	448780648978 6	29	ja.m.wikipedia.org	451773538098 6	30	ja.m.wikipedia.org	423185112608 7	1	ja.m.wikipedia.org	424109984113 7	2	ja.m.wikipedia.org	535223235640 7	3	ja.m.wikipedia.org	537992180969 7	4	ja.m.wikipedia.org	448322594045 7	5	ja.m.wikipedia.org	445007695821 7	6	ja.m.wikipedia.org	435472597765 7	7	ja.m.wikipedia.org	430814756332 7	8	ja.m.wikipedia.org	461334758600 7	9	ja.m.wikipedia.org	547280892717 7	10	ja.m.wikipedia.org	600062076779 7	11	ja.m.wikipedia.org	473057269970 7	12	ja.m.wikipedia.org	477262829644 7	13	ja.m.wikipedia.org	471436878948 7	14	ja.m.wikipedia.org	472398332176 7	15	ja.m.wikipedia.org	548724119441 7	16	ja.m.wikipedia.org	616527657391 7	17	ja.m.wikipedia.org	639723642046 7	18	ja.m.wikipedia.org	638694292375 7	19	ja.m.wikipedia.org	481106407202 7	20	ja.m.wikipedia.org	473148500657 7	21	ja.m.wikipedia.org	466261658125 7	22	ja.m.wikipedia.org	465249325390 7	23	ja.m.wikipedia.org	492041513150 7	24	ja.m.wikipedia.org	563837017528 7	25	ja.m.wikipedia.org	445503057133 7	26	ja.m.wikipedia.org	447799358192 7	27	ja.m.wikipedia.org	447035047673 7	28	ja.m.wikipedia.org	432588647022 7	29	ja.m.wikipedia.org	447436227804 7	30	ja.m.wikipedia.org	503914286671 7	31	ja.m.wikipedia.org	553840973015 8	1	ja.m.wikipedia.org	465305749921 8	2	ja.m.wikipedia.org	459082686410 8	3	ja.m.wikipedia.org	468718936637 8	4	ja.m.wikipedia.org	463721744365

Page views
To rule out bytes saved being due to a drop in page views, we checked the traffic to mobile site, in ja.wikipedia (i.e., ja.m.wikipedia.org) in the months of June and July. In June the user traffic was 585 millions and in July 587 millions. (normalized for a 30-day month). The two months are pretty much the same in terms of traffic for practical purposes so we concluded that the savings in bytes was not likely due to a macro change in traffic.