Contributors/Projects/Editing performance

Report publication date: 2 March 2018

In order to better understand the performance of the 2017 wikitext editor relative to the 2010 wikitext editor, the Editing team has collected performance metrics for these editing environments. This page describes the metrics and how they were collected, and analyses the data to recommend a course of action.

Definitions

"Time to interactive" is the amount of time it takes for the page to be open and ready for user input. At this point, you could start typing, although the page may not have all features ready for use. (For example, in the visual editor, images and parts of the toolbar might still be loading in the background.) For our purposes, this is defined as the amount of time between the user clicking an "Edit" or "Edit source" button, and the user being able to type.

"Time to fully loaded" is the amount of time it takes for the page to complete the loading process. All images, toolbars, and other assets will be fully loaded, and all functionality will be enabled. The page may be interactive before this happens. For our purposes, this is defined as the amount of time between the user clicking an "Edit" or "Edit source" button, and the page having completed loading.

We have chosen these two metrics because they best represent user-perceived performance. The time to fully loaded metric is useful but, in the modern era where additional features are often loaded by JavaScript, it does not take into account the fact that users are frequently able to begin core actions, such as typing, before the page has fully completed loading.

Methods

To collect the data, we instrumented the 2017 and 2010 wikitext editors to capture data conforming to the above definitions.

Although evaluating the visual editor's performance is outside the scope of this analysis, we also instrumented it in the same manner as the 2017 and 2010 wikitext editors, using the same metrics. This was done for reference purposes, and because the data will be useful in the future.

The data reported here was captured is from all of the live Wikimedia sites, so it represents the real experience that our users are getting, with the mix of page sizes, page complexities, different browsers, computer operating systems, network connections, user scripts, geographical locations, and so on, that current users are experiencing now. The sample size is 6.25% of all edits made using the three editing environments, in all namespaces. The initial data included in this report was collected over several weeks in December 2017 and January 2018, and represents a total of more than one million edits.

Results

No two real-world edits are exactly identical, and a single average provides very little information. To compare performance, the data set has been sorted by speed. This allows us to compare the faster (or slower) edits in each editing environment against the faster (or slower) edits in the other editing environments. Results are given below for particular percentiles. The number given at the 50th percentile (median) means that 50% of the edits using that editing environment were faster than the number given, and 50% were slower. Similarly, the number given at the 90th percentile means that 90% of edits in that editing environment were faster than this, and 10% were slower.

Time to interactive

Time to interactive	Speed to interactive (1/time)

Time to fully loaded

Time to fully loaded	Speed to fully loaded (1/time)

Data table

Performance data by percentile
Time (seconds) to interactive	1st	5th	10th	20th	25th	50th (median)	75th	80th	90th	95th	99th
Visual editor	0.634	0.878	1.068	1.353	1.487	2.358	4.203	4.978	8.1	13.168	47.328
2017 wikitext	0.421	0.567	0.662	0.829	0.925	1.562	3.068	3.592	6.674	10.939	45.869
2010 wikitext	0.547	0.717	0.827	0.999	1.081	1.565	2.554	2.974	5.020	10.625	106.66
Time (seconds) to fully loaded	1st	5th	10th	20th	25th	50th (median)	75th	80th	90th	95th	99th
Visual editor	0.635	0.879	1.069	1.355	1.488	2.360	4.205	4.980	8.104	13.168	47.332
2017 wikitext	0.422	0.568	0.663	0.830	0.926	1.563	3.071	3.589	6.673	10.896	45.871
2010 wikitext	0.693	0.899	1.034	1.247	1.348	1.953	3.206	3.738	6.246	13.103	113.008

The visual editor and 2017 wikitext editor have almost identical data for time to interactive and time to fully loaded; this is expected, because these editors presently become interactive only upon being fully loaded.

Conclusions

In half of total edits, users of either of the two wikitext editors can start typing less than 1.6 seconds after clicking the Edit button, and, for the other half of edits, they have to wait longer than that to get started. Users of the visual editor, for comparison, have to wait less than 2.4 seconds for half of their edits, and more than that for the other half of their edits.

In just over 80% of edits, the 2017 wikitext editor is fully loaded faster than the 2010 wikitext editor. In about 50% of edits, the 2017 wikitext editor is interactive faster than the 2010 wikitext editor; variance in the data made it difficult to distinguish between the two. So, generally speaking, the 2017 wikitext editor is about the same speed to become interactive, but faster to be fully loaded, than the 2010 wikitext editor.

Caveats

Assignment between conditions in this observational experiment is not completely random. Most users of the 2017 wikitext editor have made a deliberate choice to use that editing environment. We expect that users who experienced significant performance problems, within their personal computing environment, would normally stop using the 2017 wikitext editor. Similarly, we expect people who perceived the 2017 wikitext editor to be the faster option, within their personal computing environment, would normally be less likely to return to the 2010 wikitext editor.

Next steps

As a result of this analysis, the Editing team did some work on improving the time it takes for the 2017 wikitext editor to become interactive. The analysis of that work is still being conducted, and will be posted here when it is finished.