Draft namespace/Usability testing

The following document describes resources, questions, and a usability testing scenarios for enhancements to the Draft namepsace. The plan defined and a summary of the observations are available below.

Test plan

Materials

Initial designs
Blog post, to start circulating the participants form.
Prototype from redlink
Prototype from search

Goals

In these sessions, we want to test...

Understandability of draft concept. Is the concept of draft (and how it relates to an article) understood by users?
Discoverability of entry points. Are users able to start a new draft?
Awareness of draft mode. Is it clear when users are in draft mode (and how that is different from an article)?
Understanding of draft creation process. Are users able to create a draft and publish/abandon it?
Understanding of aids. Do users understand the purpose of the different tools provided?
Learn about collaboration patterns for editing. How are users collaborating to each other when editing and how they expect to be able to do so?

Pre-test questions

How was your experience editing Wikipedia (if you ever did)?
Have you tried to create a new article?
- Yes: Can you describe the process you followed for creating a new article?
  - Pay attention to entry points used (search, direct URL, red links).
  - Have you experienced a deletion?
- No: Is there any reason you think it prevents you to create a new article about any of your topics of interest?
  - Hypotheses: Not knowing you can edit, no ideas of what to contribute to, perceived lack of valuable knowledge, lack of confidence, lack of technical knowledge, lack of time to invest on that.
Can you check if there is an article about “Driving in Germany” in Wikipedia and try to start a new one? There is no need to add any real content, we are not going to save it.
When creating a Wikipedia article (or any other kind of document such as a presentation or a report for you work), do you collaborate with other users or you share your work with other users?
When you are editing, how do you identify the important aspects that need improvement?

Test scenarios

The following are our testing steps:

Intro. In this scenario, you are reading the wikipedia article about traffic lights to check if their color code is universal or changes from country to country.

Show the prototype where there is a red link that allows users to start a draft: Imagine that you are also interested in “pedestrian crossings” which are mentioned in the article. Would you be able to add some information about this topic?

[The user gets to the draft page]

Could you tell us which is the purpose of this?
Can you add some content and make it available as a wikipedia article?

Show the prototype with the search entry point (this leads to the same draft but has some comments from users): Some months later, you got a notification telling you that your article was not ready for publishing. Could you search for the “Pedestrian crossing” article to add some more information?

[The user access the draft.]
Do you have an idea on what to change to make it ready to be published?
How would you ask other editors to help you with this?
In case you have a friend which is an expert on the topic, how would you share this with him to let him participate?
If you wanted to add a link [or make some text bold] what would you do? [Not supported in prototype, but allows to check if toolbar become disconnected from content]
[For advanced users] Could you revert to the regular Article creation if you don’t want to go through the draft process?

Post-test questions

You have created a draft before the content become a real article. In which situations you think you would use drafts?
Was it clear when articles were lacking, and how to create new drafts?
Was it clear at anytime when you were working on a draft and when you were in a published article?
Do you have any issue saving, publishing, and requesting help on drafts?
In which scenarios would that be useful?
After your initial impression of the tool, do you think you’ll be more likely to create (more) new articles?

Test results

On January 2014, a round of usability testing was performed. Four users participated in the testing sessions. A summary of the test results is provided below. For the detailed observations of each test you can check the detailed results.

What worked well

Draft designs can improve the current “article for creation” (AfC) process (and similar ones). AfC process is painful because it does not take context into account (asking to check if the topic exist even when the user just searched for it), forces the user to prepare everything in advance (e.g., do you have valid references?), encourages a “read the manual before doing anything” approach, and postpones the point where the user can start working on the content. Draft designs provide a more direct access to start editing and tries to provide information when it is relevant.
Redlink popup works. The redlink popup conveys the idea that the article is lacking and provides options for article creation that users can use.
- The distinction between draft and article may not be immediately obvious especially when seen “create draft” and “create article” side by side.
  - Wording can be tweaked to better distinguish both concepts
Drafts on editing mode are distinguishable from regular articles. Users landing to a draft recognize it as such even when they just published it as an article minutes before.
- The transition from draft to article is well interpreted but changes to the draft can be better communicated:
  - A notification, would be needed to make the user aware of such event.
  - An indicator to communicate status change would be helpful. For example: “This was rejected as an article X days ago, help to improve it before it is published.”
  - The “history” indicator may be hard to discover as an entry point to check “what happened”. A link to “history” as part of the status communication can be useful.
The save-publish process works. During draft editing, the main call to action changes from “save” to “publish” depending on the context. Users were able to use those actions as expected, and when asked to publish directly, they tried to expand the “save” drop-down button to look for the publishing option.
- Direct option for publishing makes is essential in the dropdown button. Other actions such as “preview” may be useful.
Draft indicator on search works. Users were able to discover the indicator of drafts from the search results, interpret it correctly and access to the draft to improve it.

What didn’t work so well

Discussion items are not immediately visible. The list of discussion topics seems to be initially ignored. Once it is discovered, it s understood that those are suggestions for improvement from other users.
- Simplification may help to perceive is less as a noisy area.
- Progressive disclosure may help initially: Don’t show the initial automatic suggestion until the user has written a paragraph. the initial guider that announces the feature can also be postponed to that point in time.
It is hard to anticipate the effects of “Invite any editor”. It is not clear who will be contacted when requesting help to “anyone”.
- We can use more specific wording (“any interested editor”, “anyone interested in the topic”, “anyone who wish to help”...) to convey the idea that some users related to the topic will be contacted.
- Auto-completion and suggestion of closeby users (those you interact most) may be helpful here.
- We may want to separate invite (contact users) from share (distribute links outside), to focus and clarify, but it may have its problems too (forcing the user to distinguish and decide upfront).
- More clarity can be provided through all the messages about you vs. others participation. We want the users to ask for help but also to improve themselves the articles not depending too much on others.
Abandon the draft mode may not be totally clear. Although it seems to work (users looked for the discard option under the draft menu), the separation of actions between button (save, publish, preview) and draft header (discard, disable) may not be clear. We need to keep observing the discoverability of them.
- More clarity is needed for the options to discard a draft and disable draft mode.
  - We may consider showing one option at a time (if there is no content, an explicit “discard” option makes no sense)
  - Another option could be to use a generic exit, with a dialog that allows to jump to regular article creation.
  - Exiting draft mode to regular article creation should affect the default creation mode.
- Switching modes from the draft is not a big issue since users can go back to the entry point and pick a different creation mode.
More prominence needed for quality criteria. The initial summary (notable, reliable, educational) is good as a general summary, but topic-specific criteria may be needed.
- We can encourage, adding categories, and show suggestions based on them. Wikidata may help to identify if categories belong to “living people” , “companies” or other general topics with specific criteria.
- Show a warning before publishing if we detect that the article may not be ready (e.g., too short, too long paragraphs lacking formatting, unresolved elements in the conversation area...)