Contributors/Projects/Editing performance

From mediawiki.org

Report publication date: 2 March 2018

Contact: Dan Garry

In order to better understand the performance of the 2017 wikitext editor relative to the 2010 wikitext editor, the Editing team has collected performance metrics for these editing environments. This page describes the metrics and how they were collected, and analyses the data to recommend a course of action.

Definitions[edit]

"Time to interactive" is the amount of time it takes for the page to be open and ready for user input. At this point, you could start typing, although the page may not have all features ready for use. (For example, in the visual editor, images and parts of the toolbar might still be loading in the background.) For our purposes, this is defined as the amount of time between the user clicking an "Edit" or "Edit source" button, and the user being able to type.

"Time to fully loaded" is the amount of time it takes for the page to complete the loading process. All images, toolbars, and other assets will be fully loaded, and all functionality will be enabled. The page may be interactive before this happens. For our purposes, this is defined as the amount of time between the user clicking an "Edit" or "Edit source" button, and the page having completed loading.

We have chosen these two metrics because they best represent user-perceived performance. The time to fully loaded metric is useful but, in the modern era where additional features are often loaded by JavaScript, it does not take into account the fact that users are frequently able to begin core actions, such as typing, before the page has fully completed loading.

Methods[edit]

To collect the data, we instrumented the 2017 and 2010 wikitext editors to capture data conforming to the above definitions.

Although evaluating the visual editor's performance is outside the scope of this analysis, we also instrumented it in the same manner as the 2017 and 2010 wikitext editors, using the same metrics. This was done for reference purposes, and because the data will be useful in the future.

The data reported here was captured is from all of the live Wikimedia sites, so it represents the real experience that our users are getting, with the mix of page sizes, page complexities, different browsers, computer operating systems, network connections, user scripts, geographical locations, and so on, that current users are experiencing now. The sample size is 6.25% of all edits made using the three editing environments, in all namespaces. The initial data included in this report was collected over several weeks in December 2017 and January 2018, and represents a total of more than one million edits.

Results[edit]

No two real-world edits are exactly identical, and a single average provides very little information. To compare performance, the data set has been sorted by speed. This allows us to compare the faster (or slower) edits in each editing environment against the faster (or slower) edits in the other editing environments. Results are given below for particular percentiles. The number given at the 50th percentile (median) means that 50% of the edits using that editing environment were faster than the number given, and 50% were slower. Similarly, the number given at the 90th percentile means that 90% of edits in that editing environment were faster than this, and 10% were slower.

Time to interactive[edit]

Time to interactive Speed to interactive (1/time)

Time to fully loaded[edit]

Time to fully loaded Speed to fully loaded (1/time)

Data table[edit]

Performance data by percentile
Time (seconds) to interactive 1st 5th 10th 20th 25th 50th

(median)

75th 80th 90th 95th 99th
Visual editor 0.634 0.878 1.068 1.353 1.487 2.358 4.203 4.978 8.1 13.168 47.328
2017 wikitext 0.421 0.567 0.662 0.829 0.925 1.562 3.068 3.592 6.674 10.939 45.869
2010 wikitext 0.547 0.717 0.827 0.999 1.081 1.565 2.554 2.974 5.020 10.625 106.66
Time (seconds) to fully loaded 1st 5th 10th 20th 25th 50th

(median)

75th 80th 90th 95th 99th
Visual editor 0.635 0.879 1.069 1.355 1.488 2.360 4.205 4.980 8.104 13.168 47.332
2017 wikitext 0.422 0.568 0.663 0.830 0.926 1.563 3.071 3.589 6.673 10.896 45.871
2010 wikitext 0.693 0.899 1.034 1.247 1.348 1.953 3.206 3.738 6.246 13.103 113.008

The visual editor and 2017 wikitext editor have almost identical data for time to interactive and time to fully loaded; this is expected, because these editors presently become interactive only upon being fully loaded.

Conclusions[edit]

In half of total edits, users of either of the two wikitext editors can start typing less than 1.6 seconds after clicking the Edit button, and, for the other half of edits, they have to wait longer than that to get started. Users of the visual editor, for comparison, have to wait less than 2.4 seconds for half of their edits, and more than that for the other half of their edits.

In just over 80% of edits, the 2017 wikitext editor is fully loaded faster than the 2010 wikitext editor. In about 50% of edits, the 2017 wikitext editor is interactive faster than the 2010 wikitext editor; variance in the data made it difficult to distinguish between the two. So, generally speaking, the 2017 wikitext editor is about the same speed to become interactive, but faster to be fully loaded, than the 2010 wikitext editor.

Caveats[edit]

Assignment between conditions in this observational experiment is not completely random. Most users of the 2017 wikitext editor have made a deliberate choice to use that editing environment. We expect that users who experienced significant performance problems, within their personal computing environment, would normally stop using the 2017 wikitext editor. Similarly, we expect people who perceived the 2017 wikitext editor to be the faster option, within their personal computing environment, would normally be less likely to return to the 2010 wikitext editor.

Next steps[edit]

As a result of this analysis, the Editing team did some work on improving the time it takes for the 2017 wikitext editor to become interactive. The analysis of that work is still being conducted, and will be posted here when it is finished.