Reading/Multimedia/December 2015 cross-wiki upload A/B test

From mediawiki.org

In December 2015, Multimedia team ran an A/B test of four different interfaces for cross-wiki upload tool.

Background[edit]

Screenshot of the second (and final) page of cross-wiki upload dialog in VisualEditor.

Cross-wiki upload functionality was first announced on the multimedia@ list (as "Upload tool strategy change") and then developed at T91717, with the first deployment of uploads in VisualEditor happening on 2015-10-21. (Uploads in wikitext editor followed, per T115748, on 2015-10-28.) It was announced afterwards in Tech News 2015/42 and on Commons village pump.

The dialog allows any registered user to upload a self-made file under the CC BY-SA license from any Wikimedia wiki to Commons, guiding them through the process. Its interface is inspired by UploadWizard's, but much more limited (in particular, it only allows uploading one file at a time and only allows providing a simple description in one language).

After a while, it was noticed on 2015-12-07 that while the tool is successful at producing uploads (~1000 a day), many of them are copyright violations uploaded by users who didn't notice, ignored or misunderstood the warnings. T120867 was filed and we set out to design an interface providing more information to new users and more resistant to being blindly clicked through, while not too discouraging to good contributors.

For Commons editors[edit]

All new cross-wiki uploads are tagged with cross-wiki-upload and the list can be seen at Special:Log/upload?tagfilter=cross-wiki-upload on Commons. These tags are also visible in file edit history, upload log and recent changes.

Cross-wiki uploads uploaded during the A/B test are tagged with cross-wiki-upload-1 through 4. These tags are not ordinarily visible, but you can still use them for filtering. The number of files uploaded with each tag (including deleted files) can be seen at Special:Tags on Commons.

The test is not enabled on all wikis (in particular, it's not enabled on ones in languages that were not translated in time), so you may still see uploads without the numbered tags while the test is ongoing.

Implementation[edit]

The test was implemented using change I557056b8 (which implemented the new interfaces and a method to select which one the user will see) and change I90cb12c5 (which recorded the interface used for given upload server-side using change tags).

Users are bucketed into four groups by taking their user ID number modulo 4 (the test only runs for registered users, as only registered users can upload files). For testing and debugging, one can add '?uploadbucket=N' to the URL to force given interface option to appear (any upload completed this way won't count towards the bucket). The test can be enabled by setting $wgForeignUploadTestEnabled to true, and disabled without code changes by setting it back to false. To change the default after the test is over, $wgForeignUploadTestDefault can be set to the number of the winning option.

The test was enabled with change If10f3e54 (for English-language wikis, responsible of ~43% of cross-wiki uploads so far) and change I12a7ab2a (nine other languages, responsible for ~17% of uploads) and began on 2015-12-16. The test was disabled with change I13b77861 on 2015-12-29.

Timeline[edit]

  • 2015-12-16 – test began
  • 2015-12-23 – planned test end, but extended due to lower than expected number of uploads
  • 2015-12-29 – test ended
  • 2016-02-05 – results posted

Options[edit]

After brainstorming at T121021, we settled on four options:

Each of them will be seen by ~25% of users. Any given user will only see one of the options throughout the duration of the test.

Results[edit]

The following results were compiled on 2016-02-04, a little over a month after the test ended, when the number of deleted files has mostly stabilized. (Great thanks to the Commons community for reviewing the files!)

Software used to compile the numbers is available at https://github.com/MatmaRex/commons-crosswiki-uploads. Raw data it generated is available at https://phabricator.wikimedia.org/F3311649.

Summary[edit]

The questions in option 3 resulted in fewer uploads than the other ones, but the difference wasn't very large. The additional checkboxes in option 2 and the introduction in option 4 have not convinced any users to abandon their uploads.

Users apparently mostly ignored our attempts to educate them and uploaded inappropriate files anyway. Option 3 resulted in slightly better uploads by first-time uploaders, but it's not clear if the result is significant.

The tested interface options weren't very successful in improving the quality of uploads, by new users or otherwise. The upload dialog was reverted to using option 1 for now.

Detailed results[edit]

You can view the full list of uploads participating in the test (large page).

"Bad" uploads below refers to files that have been deleted, "good" uploads refers to files that have remained.

First-time uploaders[edit]

The following includes only users for whom the file upload was the very first contribution to Commons. (Contributions to other wikis weren't counted.) If the user uploaded multiple files, only of them is included.

Graph of the number of uploads done using each interface option.
Uploads done using each interface option
Interface option 1 2 3 4
Number of good uploads 242 247 220 206
Number of bad uploads 314 336 266 361
Total number of uploads 556 583 486 567
Percentage of good uploads 43.53% 42.37% 45.27% 36.33%

All users[edit]

Graph of the number of uploads done using each interface option.
Uploads done using each interface option
Interface option 1 2 3 4
Number of good uploads 506 488 375 457
Number of bad uploads 678 765 581 800
Total number of uploads 1184 1253 956 1257
Percentage of good uploads 42.74% 38.95% 39.23% 36.36%

Further work[edit]

Work on a second iteration of the test has started. Please follow T120867 for updates.