Flow/Research/Usability testing 2013-11-23

For this round of usability testing, we targeted a random set of users near the University of California, Berkeley. Most of them were students at the university. Though they all used Wikipedia on a weekly or daily basis as readers, none had ever edited before, and very few had ever seen a talk page.

Preliminary questions

How often do you use Wikipedia?
Have you ever edited Wikipedia before?
Have you ever looked at a discussion page on Wikipedia before?

Tasks attempted on the Flow prototype (on the River source article)

If you wanted to edit this page, do you know how you would do that?
If you wanted to see discussions related to this page, do you know how you would do that?
Let's say you wanted to reply to the topic "Reducing overly strong pushing of one definition over others." Can you do that now?
How about if you wanted to reply directly to one of the comments in that discussion?
Let's say you wanted to start a brand new discussion on this page. Where would you go to do that?
Overall, did you find these tasks easy or difficult?
Do you have any suggestions for improvement?
Would you be more likely to participate in discussions on Wikipedia after what you've seen today?

Overall results

All used Wikipedia frequently (at least a few times a week, many every day). None had ever edited before.
Most knew about the edit button even if they'd never used it, but only 2 even knew there was discussion system on Wikipedia, and even when told to look for it, people had a hard time finding the discussion tab.
When they finally found it, they said the discussion page looks like a place where people leave comments, ask questions, and/or offer opinions related to the article.
Nobody had trouble finding/using "start new topic" or replying to a topic in the bottom reply box, but some had trouble catching the reply link affordance for replies to replies – they said the hover made it less discoverable and suggested it be displayed all the time.
Nobody noticed or mentioned the WikiProject header templates. All the users just scrolled right past without looking.
Though there were a few confusing elements, all generally found Flow very easy to use and said commenting worked like they expected. Most said they'd be more likely to participate in discussion on Wikipedia if it looked like this.

Notes from individual tests

Mike

Late teens/early 20s

Uses Wikipedia at least once a day.
Has never edited but knows how. When looking at an article, easily finds the edit button.
Has never participated in discussions and doesn't know he could do that. He finds the discussion tab when prompted to look for it.
When asked to comment on a topic, has no trouble finding reply box.
Confused about preview button – not sure what it would do. Also confused about the mention text (User) prefilled in the comment box. When he previews the comment he's typing, he clicks on the user link in the preview and gets brought to the user page ("I didn't know there were user profiles on Wikipedia!")
Easily finds "start new topic" button and uses it.
Figures out that the number next to his comment is an IP address, but not sure why.
Overall impressions: "easy," "intuitive," is more likely to contribute to discussions.
Suggestions for improvement: have "reply" button always visible, not just on hover.

Sally

Late teens/early 20s

Uses Wikipedia "every 20 minutes" – treats it like a dictionary to look up stuff she doesn't know
Hasn't edited but has some idea how to do it and finds edit button when asked. When she opens the edit window, she's freaked out by the "scary code." "Can I write over it? I really don't know anything about coding."
Has no idea there's discussions on Wikipedia but finds the tab when prompted. Thinks that this is a place where people give their opinions or ask questions about the content, kind of like Yahoo answers.
Has no trouble replying to a topic by using the reply box. Doesn't get the IP thing at all, though.
She hovers over a username and sees "talk" and "contribs" links – asks if you "can IM someone" by clicking on "talk".
Misses the direct reply link, but finds "start topic" right away.
Overall impressions: easy to use, could see herself using this if she had a specific question about something that wasn't covered in an article.
Suggestions for improvement: discussion should be at the bottom of the page (below the article), like on other websites

Adrian

30s/40s

Uses Wikipedia often.
Doesn't edit or know about discussions (or the fact that there are users). Thinks that in order to edit, you have to go to "inspect source" and edit the HTML!
When taken to the talk page, says there's too much going on. Also says he's not the kind of guy who'd participate – would expect "somebody else to do it."
Figures out how to reply to a topic but thinks hitting enter will save.
Gets that the IP has something to do with being anonymous. When he realizes it's an IP address, he's not happy about it. "I like my privacy."
Easily finds "start new topic" affordance, but he doesn't realize that he needs to fill out a topic title to save (and the cursor automatically focuses on the comment field, not the title field), so he can't save it and gets confused/frustrated.
Overall impressions: not likely to use this, but more due to skepticism about Wikipedia's reliability, not the software.
Suggestions for improvement: have a toolbar that follows you on the page for posting new comments, and feels that discussion should be the bottom of the page, below the article, like on other websites

Tristan

Late teens/early 20s

Uses Wikipedia at least once a day.
He hasn't edited (doesn't know how) but finds edit button easily. When he opens the edit window, he says it looks like some formatting and text – "not too hard," would feel comfortable making a change.
He has looked at discussion pages before, but he's not sure how to contribute.
When he goes to the discussion page: "looks different.. better." Looks clearer, more like the kinds of comment forums he's used to.
Unlike all the other users, he spots the edit title and permalink icons to the right of the topics (he gets edit title, but not sure what permalink does).
Easily replies to topic; works as expected.
Gets the gist of the IP address – he doesn't have a user account, so this is some form of identification.
Easily finds and uses "start new topic," reply to comment, gets the mention text prefilled in the comment box ("lets you reply directly").
Overall impressions: very easy to use, something I see on most forums. "Exactly how it should be."
Suggestions for improvement: an overview of topics/table of contents, suggests Something Awful forums as example of interface he thinks is good.

Laura

Late teens/early 20s

Uses Wikipedia at least a couple times a week.
Hasn't edited but knows about the edit button and finds it on the page. Daunted when she opens the edit window (there's a "bunch of stuff... I don't know how to do this.")
Doesn't know about discussions. Even when prompted to look for it, can't find it on the page.
When she finally gets to the discussion page, she gets the gist – place where "people can comment on certain paragraphs of the article, so they can talk about any misinformation."
Finds "start topic" and reply to comment easily.
Doesn't know what the IP numbers mean.
Overall impressions: works like she expected. She'd be more likely to read discussions now. She always read the comments section on other sites to see what people have to say.
Suggestions for improvement: if someone starts discussions on one part of the article, maybe include a link to that part of the article? Also, discussions should be at the bottom – mentions online news sites like Huffington Post always have comments below article.

Kelly

Late teens/early 20s

Uses Wikipedia every time she has a paper to write (though her professors tell her not to cite it). She likes to get an overview of subjects and find sources from WP.
Didn't know that she can edit, or that there are discussions.
She finds the edit button when asked, but wouldn't feel comfortable making changes. She'd be afraid of giving incorrect information.
Has a hard time finding the discussion tab – "it should be more visible."
When she gets to the discussion page, replying to a topic and starting new topic is easy for her.
Doesn't get the IP number – "Is it my name?"
Thinks you click on the username to reply (this is how it works on BSpace, the Berkeley student course software that she uses for her classes).
Doesn't see why you'd need "preview," but gets that the text in the reply box is the person she's replying to.
Overall impressions: "I like it. It's easy." More likely to look at discussions like this; she's interested in what people are saying about the article.
Suggestions for improvement: the reply button appearing on hover is confusing for her. Thinks it should always show up.

Zara

Late teens/early 20s

Uses Wikipedia a few times a week.
Hasn't edited, but has looked at discussions. When something's controversial, looks at it to see if there's stuff she should be aware of.
Finds edit button easily but "wouldn't know what to do"
Finds discussion tab – says the page looks familiar. (Not sure she's actually looked at real Wikipedia talk pages like she claims :-P)
Finds reply box, which works as she expected.
Doesn't get the IP thing.
Doesn't see where she can start a new topic (but she seems pretty ready to be done at this point, so she doesn't really try).
No overall impressions/suggestions – in a hurry to go.