Draft namespace/Usability testing/Results

The observations include timestamps that correspond to the time in the recordings and user IDs that are preserved for tracking.

User #5

Usability testing session January 2014 - User 05

Pre-test questions

• 00:08: The user has little editing experience (“I haven’t edited too much, 3 times or something”) but finds it “easier than HTML”.
• 00:53: The user has never tried to create a new article because the lack of clarity about the process of creating a new article.
• 01:49: “I’m concerned about whether it [the new article] would get approved or not.”
• 04:05: The user tries to create a new article:
  • Searches for the topic
  • Identifies that the topic is lacking.
  • Uses the “ask for it to be created” call to action to create the new article.
  • The user lands into the article creation page and goes directly to “click here to create an article now!”.
  • The user decides to “write an article now” (instead of request for others to create it).
  • the user avoids searching again since she is sure it does not exist form search (the system does not take into account she already get there by search).
  • The user selects a category.
  • the user is asked if the subject is notable, and after som doubts decides it is.
  • The user is asked if the article has good sources. She does not know it in advance, but assumes feter some seconds she’ll have.
  • the user confirms that the submission meets the requirements.
  • The user is proposed to chose between create it as a draft or directly as an article. The user prefers to start as a draft.
  • The user forgets to type the name of the article and gets a big error when trying to create the draft.

Draft prototype

• 10:33: the user identifies that, “pedestrian crossing” is lacking, clicks on it, and immediately jumps to create a draft.
• 11:20: The user identifies that it is a draft, reads the introductory text and starts adding content.
• 15:08: The user comments that she would add “more content, references and photos” before publishing.
• 15:33: the user identifies the “save draft” action and is able to follow the saving workflow (small confusion is produced because edit summary is not functional in the prototype).
• 16:37: The user is unable to continue editing because the text became non-editable in the prototype by mistake.
• 18:22: The user is able to add more content, save and then publish. The user tried to look the option for directly published in the drop-down button.

Draft prototype (search)

• 20:43: When the user searches for the article, she identifies that “a draft is in progress”.
• 21:19: The user thinks that the published article is now a draft because “probably somebody revoked because it had not enough content to be published as an article”.
• 23:06: The user only finds the discussion topics after being asked to look for clues on what to improve. they were not discovered initially.
• 23:48: The user identifies “invite collaborators” as the way to ask other users to help.
• 25:24: The user expects “invite any editor” to contact users that have the draft in the watchlist.

Post-test questions

• 28:03: The user thinks that drafts can be useful to avoid your work to be deleted if you don’t have time to make it more complete.
• 29:36: the user mentions it is not clear how to create it a sa regular article, and also that “like the interface but it is not clear how to do fancy stuff like adding photos” (last part may be related to VE).

User #8

Usability testing session January 2014 - User 08

Pre-test questions

• 00:26: Editor for about 5 years, created 50-100 articles. Tend to draft undr user space. Initially used Article or Creation process. as an administrator, is familiar with the quality requirements.
• 02:44: The user tries to create a new article:
  • The user searches for the topic, and identifies it is lacking.
  • Overviews the results to see if the topic is under a different name.
  • “5 years ago I would have started typing, save it, and develop over time. Because the new page patrollers are much more aggressive these days, I would probably draft it on my userspace."
  • "I heard about the new draft space, I’m not sure which functionality will add for me, so I would not use it".

Draft prototype

• 05:37: The user clicks on the red link and finds the creation menu. The user finds the style consistent with guided tours. The user is a bit confused between “Start article draft” and “Start a new article”. The user suggests to reinforce that “new article” will be “live”.
• 08:10: “This is really neat!” The user identifies Visual Editor toolbar.
• 08:40: Sidebar “looks a little busy”. The user is not sure why the draft needs discussion if he is the first there. the user comments that he is used to discussions being on a separate page (through the usual tabs, which are lacking here).
• 09:32: The user is intrigued by “invite collaborators” option, and wonders if that will be using a notification.
• 09:53: the user notices the welcome guider for drafts and considers that the new article requirements are well laid out and constitutes a good summary of complex policies.
• 10:47: The user adds some text, and finds the save button. Tries to see other options, expects info about “where it will be saved”, publish or delete. Suggests the use of a notification to indicate how to get back to drafts.
• 13:32: The user gets the save confirmation. From the “ready to publish” guider, it is not clear which is the process. the “learn how to improve your draft” button is acknowledged as helpful, but more clarity would be appreciated.
• 13:43: The user is able to publish the draft as an article, and reads the post-publish message. The user mentions that he gets the feeling that “someone else is going to improve this”. The user thinks that users should be encouraged to be bold (which is not in conflict with collaboration).
• 16:54: “It is really neat to see Visual Editor and Guided Tours working together. It looks like a real website.”

Draft prototype (search)

• 18:32: The user identifies the draft information on search. The user notices the assumption that each topic has one draft by default (on userspace drafts, they have more).
• 19:32: The user notices that other users participates and there are now 3 co-editors.
• 19:57: The user explores the “invite collaborators” feature. Finds it “cool” to be able to invite by email., but founds “scary” the option to invite “any” editor.
• 21:16: The user considers that the article became a draft again because “my article was deleted and I’m re-joining a different draft”. Only when the possibility of being “moved” is mentioned, the user thinks it is possible that this is what happened and mentions that would like to be informed about it (a notification “would be cool”).
• 22:55: the user expects the “invite collaborators” to be based on editors joining to categories. The user likes the idea of “to put a general call for help”. When explained, the user finds that looking for editors of related topics “ is a totally reasonable solution”. A more clear wording is desired to communicate this (“ask related editors”...). Auto-completion is expected for writing user names.
• 27:41: When asked to go back to regular article creation, the first try is to the draft indicator menu (where it should be, but is lacking on the prototype). the second option was to go back to search and create a new article from the article creation options provided. the user misses the availability of a search box in the prototype.
• 29:29: The user thinks it makes sense to have separate the discard from save options, but is confused between “discard” and “disable draft mode”.
• 32:40: The user finds the suggestions to be useful.
• 34:03: the user is hesitant to use it himself since he is used to his current way to draft. But he will be directing new editors to the draft namespace since “invite collaborators” and “discussion” seem interesting features (since “many new editors had no idea talk pagesexist”).
• 35:27: The user finds the draft namespace clearly identifiable because of the orange bar and the draft sidebar.

User #10

Usability testing session January 2014 - User 10

Pre-test questions

• 00:16: The user is a long time contributor of French Wikipedia, but does not consider himself a big editor because he tries to focus on helping new users to edit. He has written several article (one featured and one quality article) but has not time to edit continuously.
• 01:30: The user creates articles by translating them from English, but also from scratch on French Wikipedia. The user starts with a stub and tries to look for sources to expand it: “one sentence, one source”.
• 02:42: the user finds topics to write about based on twitter or newspapers. The user starts an article to help other people that want to complete it.
• 04:52: the user starts by searching and clicks on the red link to start an article.

Draft prototype test

• 06:27: the user accesses the red link and finds the option to create a new article.
• 07:12: The user accesses the draft option. The first thing noticed is “invite collaborators”. The user thinks he would not use the feature since he has no one to invite.
• 09:11: The user is able to contribute text and save it, add more content, and publish it.
• 11:23: The user does not noticed the discussion area initially, but identifies it as “pending actions”.
• 12:38: The user tries to add references by interacting with the VE toolbar (not working in prototype).
• 13:49: the user sees not much difference between a draft and a real article. However, he mentions he would be not confused but had a lack of information.

Draft prototype (search)

• 15:32: On search results, the user is able to identify that there is a draft. The user identifies that 3 people contributed to the draft and added comments.
• 16:52: To ask for help, the first reaction is to interact with the users that left suggestions.
• 17:41: The user assumed that “invite collaborators” refers to people already editing Wikipedia, not external.
• 18:01: “invite any editor” is strange because it seems that “I can invite everybody on Wikipedia”. the user suggests to invite people that contributed to related topics.
• 20:12: The user hopes that drafts help people to make better articles. The user thinks the discussion feature can be helpful: “I expect people to create good articles with the discussion feature”.

Post-test questions

• 21:41: the user thinks that advanced users helping new contributors is key for drafts to work. If drafts stay for long, and no one writes in their drafts, people will not be happy”. The user comments that on French Wikipedia they have a problems with the review process of drafts (users just want their draft to be published).
• 23:42: When reviewing drafts usually the user checks: if there are references, that the matter of the subject is clear, correct wikisyntax, it is neutral.
• 25:54: Different kinds of articles (e.g., actors, living people, companies) have different criteria (e.g., participation in many movies). It would be useful to display such criterias.
• 29:41: The user checks the initial guidelines (notable, reliable, and educational) and thinks that is “a good start”, but access to “learn more” may be helpful but “nobody reads”. “People forgets Wikipedia is an encyclopedia, they think it s a free website where you can write what you think about what you like”.

User #16

Usability testing session January 2014 - User 16

Pre-test questions

• 01:45: The user has not done much editing wikipedia, but has been maintaining a wiki on his work. The user participated as a reader, and edited mostly articles related to his company.
• 03:28: The user “feels bad” for not contributing. Lack of time could be a cause, but the main one is “reluctance in fear of editing community” and lack of knowledge of the rules of wikipedia.

Draft prototype

• 06:01: The user access the red link and finds the pop-up info to be “realyl nice”. Then the user explores options for article creation.
• 06:46: The user enters the draft editor. Initially he assumes that saving draft will make it to Wikipedia.
• 09:08: When asked to publish, tries to expand the “save draft” to find a wy to save as an article and not a draft. Since it is not working on the prototype, the user expects to be able to save and later publish it in some way.
• 11:23: The “ready for publishing?” tooltip helps the user to discover the publishing action. The user is able to go through the save and publish article.

Draft prototype (search)

• 18:37: When searching, the user identifies that a draft is lacking and access the draft.
• 19:16: The user access the draft, and notices the differences. The user looks for the history to check what has happened with the draft, but cannot find how to access it.
• 20:46: the user identifies the discussion panel and the number of co-editors. the user sees identifies from the discussion, the items to be improved. “Its pretty obvious”.
• 23:58: The user identifies several ways to ask help on how to improve: invite contributors, contact users that participated in the conversation, and accessing the help icon.
• 24:53: For “invite collaborators” the user expects to contact someone he knows.
• 26:39: The user identifies the different ways to share, but tries to look in the “configure more ways to share” to find some “internal way to flag it and get other editors to see it”. Functionality which is already provided by the “ask any editor” option.
• 28:33: The user thinks that drafts will help him to participate more. Since it “provides a way to start without getting me deleted and get support from other editors”.
• 29:35: The difference between articles and drafts seem “very clear”. The indicator on the left and the action to “save as draft/publish” makes it easy to identify. The user mentions that he is not used to the visual Editor layout.