Reading/Multimedia/December 2015 cross-wiki upload A/B test

In December 2015, Multimedia team ran an A/B test of four different interfaces for cross-wiki upload tool.

Background

Screenshot of the second (and final) page of cross-wiki upload dialog in VisualEditor.

Cross-wiki upload functionality was first announced on the multimedia@ list (as "Upload tool strategy change") and then developed at T91717, with the first deployment of uploads in VisualEditor happening on 2015-10-21. (Uploads in wikitext editor followed, per T115748, on 2015-10-28.) It was announced afterwards in Tech News 2015/42 and on Commons village pump.

The dialog allows any registered user to upload a self-made file under the CC BY-SA license from any Wikimedia wiki to Commons, guiding them through the process. Its interface is inspired by UploadWizard's, but much more limited (in particular, it only allows uploading one file at a time and only allows providing a simple description in one language).

After a while, it was noticed on 2015-12-07 that while the tool is successful at producing uploads (~1000 a day), many of them are copyright violations uploaded by users who didn't notice, ignored or misunderstood the warnings. T120867 was filed and we set out to design an interface providing more information to new users and more resistant to being blindly clicked through, while not too discouraging to good contributors.

For Commons editors

All new cross-wiki uploads are tagged with cross-wiki-upload and the list can be seen at Special:Log/upload?tagfilter=cross-wiki-upload on Commons. These tags are also visible in file edit history, upload log and recent changes.

Cross-wiki uploads uploaded during the A/B test are tagged with cross-wiki-upload-1 through 4. These tags are not ordinarily visible, but you can still use them for filtering. The number of files uploaded with each tag (including deleted files) can be seen at Special:Tags on Commons.

The test is not enabled on all wikis (in particular, it's not enabled on ones in languages that were not translated in time), so you may still see uploads without the numbered tags while the test is ongoing.

Implementation

The test was implemented using change I557056b8 (which implemented the new interfaces and a method to select which one the user will see) and change I90cb12c5 (which recorded the interface used for given upload server-side using change tags).

Users are bucketed into four groups by taking their user ID number modulo 4 (the test only runs for registered users, as only registered users can upload files). For testing and debugging, one can add '?uploadbucket=N' to the URL to force given interface option to appear (any upload completed this way won't count towards the bucket). The test can be enabled by setting $wgForeignUploadTestEnabled to true, and disabled without code changes by setting it back to false. To change the default after the test is over, $wgForeignUploadTestDefault can be set to the number of the winning option.

The test was enabled with change If10f3e54 (for English-language wikis, responsible of ~43% of cross-wiki uploads so far) and change I12a7ab2a (nine other languages, responsible for ~17% of uploads) and began on 2015-12-16. The test was disabled with change I13b77861 on 2015-12-29.

Timeline

2015-12-16 – test began
2015-12-23 – planned test end, but extended due to lower than expected number of uploads
2015-12-29 – test ended
2016-02-05 – results posted

Options

After brainstorming at T121021, we settled on four options:

Option 1: The original form, with a single checkbox.
Option 2: Four checkboxes, each with a label explaining one facet of the requirements (own work; no pictures of copyrighted work; educational/useful; irrevocably released as CC BY-SA).
Option 3: Some Yes/No questions structured so that 'Yes' is not always the right answer to continue uploading.
Option 4: Longer introduction before a single checkbox (as in option 1), with examples of good and unacceptable content.

Each of them will be seen by ~25% of users. Any given user will only see one of the options throughout the duration of the test.

Results

The following results were compiled on 2016-02-04, a little over a month after the test ended, when the number of deleted files has mostly stabilized. (Great thanks to the Commons community for reviewing the files!)

Software used to compile the numbers is available at https://github.com/MatmaRex/commons-crosswiki-uploads. Raw data it generated is available at https://phabricator.wikimedia.org/F3311649.

Summary

The questions in option 3 resulted in fewer uploads than the other ones, but the difference wasn't very large. The additional checkboxes in option 2 and the introduction in option 4 have not convinced any users to abandon their uploads.

Users apparently mostly ignored our attempts to educate them and uploaded inappropriate files anyway. Option 3 resulted in slightly better uploads by first-time uploaders, but it's not clear if the result is significant.

The tested interface options weren't very successful in improving the quality of uploads, by new users or otherwise. The upload dialog was reverted to using option 1 for now.

Detailed results

You can view the full list of uploads participating in the test (large page).

"Bad" uploads below refers to files that have been deleted, "good" uploads refers to files that have remained.

First-time uploaders

The following includes only users for whom the file upload was the very first contribution to Commons. (Contributions to other wikis weren't counted.) If the user uploaded multiple files, only of them is included.

Graph of the number of uploads done using each interface option.

Uploads done using each interface option
Interface option	1	2	3	4
Number of good uploads	242	247	220	206
Number of bad uploads	314	336	266	361
Total number of uploads	556	583	486	567
Percentage of good uploads	43.53%	42.37%	45.27%	36.33%

All users

Graph of the number of uploads done using each interface option.

Uploads done using each interface option
Interface option	1	2	3	4
Number of good uploads	506	488	375	457
Number of bad uploads	678	765	581	800
Total number of uploads	1184	1253	956	1257
Percentage of good uploads	42.74%	38.95%	39.23%	36.36%

Further work

Work on a second iteration of the test has started. Please follow T120867 for updates.