Prediction
editRemoving navboxes for a quality page such as Barack Obama should:
- drop the number of bytes we ship to users
- drop the fully load time
- increase the time to first byte (TTFB) from a clear cache due to the time needed for the MobileFormatter to transform the parser output
- possibly drop the first render time due to the reduction in HTML, but should not make it worse.
Method
editMobileFrontend has a library called MobileFormatter which extends the HtmlFormatter in core. We used this to strip any elements in the HTML with the class navbox. On various good quality articles the size of this HTML is significant, e.g it accounts for 15% of all HTML in the Barack Obama article.
The configuration was updated and we observed the first render and fully loaded time before and after the change.
A script was written that would calculate the median and average of values before the change during a specified period of time and after the change during the same specified period of time for a specified article (Barack Obama) on an emulated 2G connection. For example purposes, if the configuration change was made on the 15th January 2016 at midnight we would take the median of the values for a 7 day period before the change and a 7 day period after the change e.g. all test results between 10th-15th January 2016 compared with all test results 15th-20th January 2016.
The config changes were done sequentially and results were measured
- On the beta cluster in the mobile beta channel. The change went into effect on 3rd February 2016 at 21:00
- On the beta cluster in the stable channel. The change went into effect on 4th February 2016 at 22:16
- On the production cluster (in the beta channel). The change went into effect on 9th February 2016 at 22:00
The commands used were:
# beta cluster (beta)
node wptreporter.js "webpagetest.enwiki-bc-mobile-2gslow.anonymous.Barack_Obama.us-east-1.Google_Chrome-emulateMobile.firstView" 3 2 2016 21 0 "" 7
# beta cluster (stable)
node wptreporter.js "webpagetest.enwiki-bc-mobile-beta-2gslow.anonymous.Barack_Obama.us-east-1.Google_Chrome-emulateMobile.firstView" 4 2 2016 22 16 "" 7
# production cluster (beta)
node wptreporter.js "webpagetest.enwiki-mobile-beta-2gslow.anonymous.Barack_Obama.us-east-1.Google_Chrome-emulateMobile.firstView" 9 2 2016 22 0 "" 7
Results
editBeta cluster
editBeta
edit| Property | Before (avg) | after (avg) | Delta (Avg) | % decrease (Avg) | Before (median) | After (median) | Delta (median) | % decrease (median) |
|---|---|---|---|---|---|---|---|---|
| html.bytes | 192215.4 | 160596.5 | 31618.9 | 16.45% | 192216.0 | 156159.0 | 36057.0 | 18.76% |
| TTFB.median | 3912.8 | 3915.3 | -2.6 | -0.07% | 3911.0 | 3912.0 | -1.0 | -0.03% |
| render.median | 5811.6 | 5730.5 | 81.1 | 1.39% | 5884.0 | 5884.0 | 0.0 | 0.00% |
| fullyLoaded.median | 24939.9 | 25009.3 | -69.5 | -0.28% | 24226.0 | 25176.5 | -950.5 | -3.92% |
Stable
edit| Property | Before (avg) | after (avg) | Delta (Avg) | % decrease (Avg) | Before (median) | After (median) | Delta (median) | % decrease (median) |
|---|---|---|---|---|---|---|---|---|
| html.bytes | 181695.5 | 156227.1 | 25468.4 | 14.02% | 192157.5 | 156211.0 | 35946.5 | 18.71% |
| TTFB.median | 11904.0 | 13379.3 | -1475.3 | -12.39% | 12952.5 | 14403.0 | -1450.5 | -11.20% |
| render.median | 15527.0 | 16964.3 | -1437.3 | -9.26% | 16688.0 | 18087.5 | -1399.5 | -8.39% |
| fullyLoaded.median | 33748.3 | 35795.5 | -2047.2 | -6.07% | 35999.0 | 36473.5 | -474.5 | -1.32% |
Production
editBeta
edit| Property | Before (avg) | after (avg) | Delta (Avg) | % decrease (Avg) | Before (median) | After (median) | Delta (median) | % decrease (median) |
|---|---|---|---|---|---|---|---|---|
| html.bytes | 188968.0 | 156178.0 | 32790.1 | 17.35% | 189029.0 | 156007.5 | 33021.5 | 17.47% |
| TTFB.median | 4947.8 | 5515.0 | -567.3 | -11.46% | 5252.5 | 5582.0 | -329.5 | -6.27% |
| render.median | 10629.4 | 8486.0 | 2143.5 | 20.17% | 11789.0 | 7568.0 | 4221.0 | 35.80% |
| fullyLoaded.median | 24248.3 | 24171.6 | 76.7 | 0.32% | 23874.5 | 23956.0 | -81.5 | -0.34% |
Analysis
editFor HTML bytes the results were rather predictable. It's worth noting that the production version of the Barack Obama is constantly changing (it's a wiki!) however the beta cluster article is mostly static. The results for bytes saved were relatively consistent.
TTFB, first render and fully loaded time all took a hit in the beta cluster. All showed a negative impact post change.
It was only in the production beta channel where results seemed to match the prediction,- a minor reduction in fully loaded time was observed, first render seemed to improve and TTFB increased for average delta and median delta.
That said, on the 12th February, it came to my attention that another experiment was in progress merged on Wednesday 10th which more than likely impacted results.
It's possible that the results on production stable may have been accurate. By reducing TTFB we may have also reduced render time since the numbers of milliseconds change are so similar. The fact fully loaded time increased however is hard to explain.
Conclusions
edit- The webpage test scripts running on the beta cluster are good for predicting changes in bytes we ship to users.
- Given the fact these tests were not run in isolated conditions (the beta cluster is constantly getting new code enabled on it, rather than once a week like the stable cluster e.g. we were updating the language overlay in parallel) I would suggest running tests in stable and beta modes on the beta cluster at the same time rather than doing them sequentially given that generally their fluctuations are similar. The beta cluster is not an isolated environment. It's used for manual and automated testing of all of WMF's deployed extensions – probably more. There's no one time when the cluster isn't under some load. In constrast to this, on production the majority of requests should go through cache.
- When measuring bytes, we should ensure the test article remains protected and unedited (we may want to setup a test article with the same content elsewhere to minimise changes to the article)
- The larger the period of time we have to measure before and after the more accurate our data will be.
- The beta cluster is not a reliable testing environment for predicting exact values for improvements in production, it however could in theory be used to predict extent of changes e.g. big change or minor change.
- We may want to use a dedicated test instance to assess changes in future.
- The production cluster can only be a reliable testing environment if we do better to coordinate changes with other times, particular the performance team.
Next steps
edit- We should apply this test to the production stable environment to assess whether there is any correlation between beta cluster / production beta channel and production stable.
- We should consider running tests on a dedicated instance outside the beta cluster and see if they give us better results.
- We need a dedicated performance cluster.