Reading/Web/Performance/How to test static pages

Setup test data

First things first you'll need to get something to test. I suggest setting up a staging environment with MobileFrontend installed. Export the pages you want from English Wikipedia and import them into your staging wiki. Where you want to alter content, create copies of the articles with the desired behaviour (e.g. remove all content post the lead section).

Run tests in parallel

Using webpagetest api and your own API key it's pretty easy to write your own script in node to generate a list of JSON that you can easily feed the array of json urls into a reporting tool.

API usage is capped so it may be better to use the Wikimedia instance, but this is a useful technique for generating results fast. reading-web-performance-test.reading-web-staging.eqiad.wmflabs is set up for carrying out such tests. Amazon EC2 takes about 10 mins to fire up an instance on us-west-1. If the instance is not used for more than an hour, it will be shut down. 10 pages, at 9 runs each, takes about 30 mins to run (on a cable connection).

On Wikimedia's WebPageTest instance

The WebPageTest library from the performance team is a useful tool for testing static pages.

Generated a sample in the scripts/batch folder in the current form:

--webPageTestKey <%WMF_WPT_KEY> --first true --bandwidthDown 35000 --bandwidthUp 32000 --latency 1300 --timeout 2400 --webPageTestHost wpt.wmftest.org --median SpeedIndex --connectivity <%WPT_CONNECTIVITY> --location us-west-1:Chrome --label firefox --runs <%WPT_RUNS> --reporter csv --file result.csv https://jonrobson.me.uk/T113649/0-a.html

--webPageTestKey <%WMF_WPT_KEY> --first true --bandwidthDown 35000 --bandwidthUp 32000 --latency 1300 --timeout 2400 --webPageTestHost wpt.wmftest.org --median SpeedIndex --connectivity <%WPT_CONNECTIVITY> --location us-west-1:Chrome --label firefox --runs <%WPT_RUNS> --reporter csv --file result.csv https://jonrobson.me.uk/T113649/0-b.html

--webPageTestKey <%WMF_WPT_KEY> --first true --bandwidthDown 35000 --bandwidthUp 32000 --latency 1300 --timeout 2400 --webPageTestHost wpt.wmftest.org --median SpeedIndex --connectivity <%WPT_CONNECTIVITY> --location us-west-1:Chrome --label firefox --runs <%WPT_RUNS> --reporter csv --file result.csv https://jonrobson.me.uk/T113649/0-c.html

--webPageTestKey <%WMF_WPT_KEY> --first true --bandwidthDown 35000 --bandwidthUp 32000 --latency 1300 --timeout 2400 --webPageTestHost wpt.wmftest.org --median SpeedIndex --connectivity <%WPT_CONNECTIVITY> --location us-west-1:Chrome --label firefox --runs <%WPT_RUNS> --reporter csv --file result.csv https://jonrobson.me.uk/T113649/1-a.html

--webPageTestKey <%WMF_WPT_KEY> --first true --bandwidthDown 35000 --bandwidthUp 32000 --latency 1300 --timeout 2400 --webPageTestHost wpt.wmftest.org --median SpeedIndex --connectivity <%WPT_CONNECTIVITY> --location us-west-1:Chrome --label firefox --runs <%WPT_RUNS> --reporter csv --file result.csv https://jonrobson.me.uk/T113649/1-b.html

--webPageTestKey <%WMF_WPT_KEY> --first true --bandwidthDown 35000 --bandwidthUp 32000 --latency 1300 --timeout 2400 --webPageTestHost wpt.wmftest.org --median SpeedIndex --connectivity <%WPT_CONNECTIVITY> --location us-west-1:Chrome --label firefox --runs <%WPT_RUNS> --reporter csv --file result.csv https://jonrobson.me.uk/T113649/1-c.html

Set up environment variables to do a single test run. Tests in a batch file will be run sequentially.

export WMF_WPT_KEY=ask peter h for one (phedenskog in #wikimedia-mobile)
export WPT_LOCATION=us-west-1:Chrome
export WPT_RUNS=1
export WPT_CONNECTIVITY=2G

When ready to run

node bin/index.js --batch scripts/batch/<your batch file>

Given that tests run sequentially youcan terminate it right away and check the correct location/connectivity settings are being used by visiting the test history page

When you are sure everything is working as expected, you will want to bump up the WPT_RUN value. Use an odd number to get a more useful median. 5,7 or 9 are suggested.

Issues

Note at time of writing you may need to do the additional following things:

• URLS for equivalent webpagetest summary URL is not present in URL. Hoping Peter will add it. It helps debugging. In the meantime you can find your test under http://wpt.wmftest.org/testlog.php
• IF 2G profile is unavailable you may need to manually define it in the batch file.
• us-east-1 is used by Jenkins, so use us-west-1 in order not to compete with Jenkins for resources.
• Use Chrome because Firefox reports wrong time to first byte.

See also

• WebPageTest