Reading/Multimedia/December 2015 cross-wiki upload A/B test
In December 2015, Multimedia team ran an A/B test of four different interfaces for cross-wiki upload tool.
Background
editCross-wiki upload functionality was first announced on the multimedia@ list (as "Upload tool strategy change") and then developed at T91717, with the first deployment of uploads in VisualEditor happening on 2015-10-21. (Uploads in wikitext editor followed, per T115748, on 2015-10-28.) It was announced afterwards in Tech News 2015/42 and on Commons village pump.
The dialog allows any registered user to upload a self-made file under the CC BY-SA license from any Wikimedia wiki to Commons, guiding them through the process. Its interface is inspired by UploadWizard's, but much more limited (in particular, it only allows uploading one file at a time and only allows providing a simple description in one language).
After a while, it was noticed on 2015-12-07 that while the tool is successful at producing uploads (~1000 a day), many of them are copyright violations uploaded by users who didn't notice, ignored or misunderstood the warnings. T120867 was filed and we set out to design an interface providing more information to new users and more resistant to being blindly clicked through, while not too discouraging to good contributors.
For Commons editors
editAll new cross-wiki uploads are tagged with cross-wiki-upload
and the list can be seen at Special:Log/upload?tagfilter=cross-wiki-upload on Commons. These tags are also visible in file edit history, upload log and recent changes.
Cross-wiki uploads uploaded during the A/B test are tagged with cross-wiki-upload-1
through 4
. These tags are not ordinarily visible, but you can still use them for filtering. The number of files uploaded with each tag (including deleted files) can be seen at Special:Tags on Commons.
The test is not enabled on all wikis (in particular, it's not enabled on ones in languages that were not translated in time), so you may still see uploads without the numbered tags while the test is ongoing.
Implementation
editThe test was implemented using change I557056b8 (which implemented the new interfaces and a method to select which one the user will see) and change I90cb12c5 (which recorded the interface used for given upload server-side using change tags).
Users are bucketed into four groups by taking their user ID number modulo 4 (the test only runs for registered users, as only registered users can upload files). For testing and debugging, one can add '?uploadbucket=N' to the URL to force given interface option to appear (any upload completed this way won't count towards the bucket). The test can be enabled by setting $wgForeignUploadTestEnabled
to true
, and disabled without code changes by setting it back to false
. To change the default after the test is over, $wgForeignUploadTestDefault
can be set to the number of the winning option.
The test was enabled with change If10f3e54 (for English-language wikis, responsible of ~43% of cross-wiki uploads so far) and change I12a7ab2a (nine other languages, responsible for ~17% of uploads) and began on 2015-12-16. The test was disabled with change I13b77861 on 2015-12-29.
Timeline
edit- 2015-12-16 – test began
- 2015-12-23 – planned test end, but extended due to lower than expected number of uploads
- 2015-12-29 – test ended
- 2016-02-05 – results posted
Options
editAfter brainstorming at T121021, we settled on four options:
-
Option 1: The original form, with a single checkbox.
-
Option 2: Four checkboxes, each with a label explaining one facet of the requirements (own work; no pictures of copyrighted work; educational/useful; irrevocably released as CC BY-SA).
-
Option 3: Some Yes/No questions structured so that 'Yes' is not always the right answer to continue uploading.
-
Option 4: Longer introduction before a single checkbox (as in option 1), with examples of good and unacceptable content.
Each of them will be seen by ~25% of users. Any given user will only see one of the options throughout the duration of the test.
Results
editThe following results were compiled on 2016-02-04, a little over a month after the test ended, when the number of deleted files has mostly stabilized. (Great thanks to the Commons community for reviewing the files!)
Software used to compile the numbers is available at https://github.com/MatmaRex/commons-crosswiki-uploads. Raw data it generated is available at https://phabricator.wikimedia.org/F3311649.
Summary
editThe questions in option 3 resulted in fewer uploads than the other ones, but the difference wasn't very large. The additional checkboxes in option 2 and the introduction in option 4 have not convinced any users to abandon their uploads.
Users apparently mostly ignored our attempts to educate them and uploaded inappropriate files anyway. Option 3 resulted in slightly better uploads by first-time uploaders, but it's not clear if the result is significant.
The tested interface options weren't very successful in improving the quality of uploads, by new users or otherwise. The upload dialog was reverted to using option 1 for now.
Detailed results
editYou can view the full list of uploads participating in the test (large page).
"Bad" uploads below refers to files that have been deleted, "good" uploads refers to files that have remained.
First-time uploaders
editThe following includes only users for whom the file upload was the very first contribution to Commons. (Contributions to other wikis weren't counted.) If the user uploaded multiple files, only of them is included.
Interface option | 1 | 2 | 3 | 4 |
---|---|---|---|---|
Number of good uploads | 242 | 247 | 220 | 206 |
Number of bad uploads | 314 | 336 | 266 | 361 |
Total number of uploads | 556 | 583 | 486 | 567 |
Percentage of good uploads | 43.53% | 42.37% | 45.27% | 36.33% |
All users
editInterface option | 1 | 2 | 3 | 4 |
---|---|---|---|---|
Number of good uploads | 506 | 488 | 375 | 457 |
Number of bad uploads | 678 | 765 | 581 | 800 |
Total number of uploads | 1184 | 1253 | 956 | 1257 |
Percentage of good uploads | 42.74% | 38.95% | 39.23% | 36.36% |
Further work
editWork on a second iteration of the test has started. Please follow T120867 for updates.