Russian Wikipedia, a large sized wiki, was the third and final test wiki to have both lazy loaded references and images enabled. It was deployed on 21st July at 4.25pm PST via SWAT window. The intention was to allow comparison with the large sized Japanese wiki which had lazy loading images enabled but not lazy loaded references enabled to inform whether it was worth pursuing lazy loaded references.
Performance Impact
editSanity checking (10 days)
editTo get a general sense of impact after 10 days of the change being live it was possible to compare 10 days prior to the change with 10 days after the change. Using the following SQL query, it was possible to take a database dump of all NavigationTiming events logged for 10 days preceding the change and 10 days following the change.
select * from NavigationTiming_15485142 where wiki = 'ruwiki' and event_mobileMode = 'stable' and timestamp > 20160711002500
This SQL query gave a sample of 167844 rows of data (5 times the amount for the corresponding test in Thai Wikipedia in half the time). With the result of the following query, using a Python script the following command was run
python navtimingcsv.py 10dayRussian.tsv 20160721002500 10
Fully loaded
edit| Label | Sample Size | 95th percentile | median |
|---|---|---|---|
| Before change | 50651 | 18282.5 | 2969.0 |
| After change | 51508 | 16533.2 | 2795.0 |
| Before change (anons) | 50601 | 18273.0 | 2970.0 |
| After change (anons) | 51458 | 16544.15 | 2795.0 |
| Before change (http2) | 36888 | 13852.6 | 2577.0 |
| After change (http2) | 37708 | 12877.65 | 2437.0 |
| Before change (http1) | 13763 | 28869.7 | 4381.0 |
| After change (http1) | 13800 | 25066.3 | 4200.0 |
First paint
edit| Label | Sample Size | 95th percentile | median |
|---|---|---|---|
| Before change | 28564 | 7597.0 | 1534.0 |
| After change | 28870 | 7489.5 | 1487.0 |
DomInteractive
edit| Label | Sample Size | 95th percentile | median |
|---|---|---|---|
| Before change | 50651 | 8199.5 | 1219.0 |
| After change | 51508 | 7940.9 | 1140.0 |
Performance Impact (28 days later)
editFully loaded
edit| Label | Sample Size | 95th percentile | median |
|---|---|---|---|
| Before change | 134448 | 18085.3 | 2943.0 |
| After change | 138351 | 16065.5 | 2758.0 |
| Before change (anons) | 134341 | 18083.0 | 2944.0 |
| After change (anons) | 138209 | 16073.6 | 2758.0 |
| Before change (http2) | 97781 | 13754.0 | 2550.0 |
| After change (http2) | 102257 | 12732.8 | 2404.0 |
| Before change (http1) | 36667 | 28191.0 | 4397.0 |
| After change (http1) | 36094 | 24719.0 | 4184.0 |
First paint
edit| Label | Sample Size | 95th percentile | median |
|---|---|---|---|
| Before change | 75418 | 7534.3 | 1527.0 |
| After change | 78208 | 7566.0 | 1459.0 |
DomInteractive
edit| Label | Sample Size | 95th percentile | median |
|---|---|---|---|
| Before change | 134448 | 8156.65 | 1201.0 |
| After change | 138351 | 7921.0 | 1123.0 |
Conclusions
editAfter almost a month of running this experiment it's clear that lazy loading references has minor impact on first paint or DomInteractive, but it does appear to improve things, but it could be argued this is due to lazy loading images. That said the performance improvements on Japanese Wiki were less significant. See Reading/Web/Lazy loading of images on Japanese Wikipedia#Analysis of 30 days of data.
The impact on fully loaded time is more significant on http1 than http2 (improving the 95th percentile by 4 seconds).
More analysis around the data savings at play here are needed.