Flow/Research/User Test Data

This page describes the results of remote usability testing related to the Flow project. For in-person testing notes and results, see Flow/Team/Meetings/2013-10-15 Flow workshop and Flow/Research/Usability testing 2013-11-23.

Flow user tests

Test: Navigating topics on board (collapse/expand)

Introduction

Background: You're helping us test out an early version of some new discussion software on Wikipedia, the world's largest free encyclopedia. Don't worry about messing anything up – this is just a prototype, not the real site! Mindset: You're reading about rivers on Wikipedia, and you decide to look at the discussion page to see what people have said about the topic. You have some ideas about how to improve this topic on Wikipedia and might want to jump into the discussion...

Tasks

Navigate to the starting URL (http://ee-flow.wmflabs.org/wiki/River_source). Take a quick moment to look at this page and talk through what you're seeing.
If you've scrolled down to read the page, go back up to the top and click on the "discussion" tab in the top left corner. (If you can't find it, follow this link). Talk through what you're seeing – what do you think this page is for?
Let's say you want to see an overview of all the discussion topics here. Without leaving this page, can you find a way to do that? (If you're still stuck after 2-3 minutes, it's okay – just move onto the next task.)
If you haven't found them already, try clicking the 3 small icons above the first discussion ("Introduction misleading") on the right-hand side. Pick a view that you like and browse the discussions on the page, talking through what you see. (Take 2-3 min)
Navigate to the discussion titled "Lake?" Let's say you have something to add to this discussion – just pretend you're an expert. Go ahead and try to participate in the discussion (again, don't worry about messing anything up!).
Now say you want to start a brand new discussion here. Can you find how to do that? Go ahead and try it if you do. Did this work like you expected? If you can't find how to do this in 1-2 minutes, you can skip the next task.
Pretend there's a typo in your new topic title – can you find how to edit it?

Questions

Have you ever looked at discussion (sometimes called "talk pages") on Wikipedia? If so, how does what you tested today compare to those pages?
Do you look at discussions or comments sections online on other sites? How does what you tested just now compare to that? Did it remind you of those other sites, or did it look very different?
Was there anything you found especially confusing or frustrating?
Would you be okay with us sharing this video with our editor community? We build features in partnership with them, and your experiences can help us and them see what we need to improve in our software.

Results

Beta4site

Beta4site

51-yr-old male, "beginner" online experience, has never looked at a talk page on Wikipedia

Test notes

Confused by WikiProject header stuff
Overwhelmed by the all open view, but he seems to really like the middle (collapsed) view - prefers over small view and open view, makes it easier for him to browse
When asked how he would get an overview of all the topics on this page, easily finds the 3 buttons that toggle the view of the board (chooses the smallest view first, then tries the middle view).
Clicks on "10 comments" to expand discussion when it's closed (which doesn't work).
Has no trouble collapsing/expanding throughout the rest of the test to accomplish other tasks (participate in a discussion, start a new topic).
Uses bottom comment box to "participate in the discussion"
Tries to edit title by clicking on it
Does a lot of random clicking around (which expands/collapses stuff unnecessarily) before finding action menu & edit title

Question answers

Has never seen a talk page.
Uses discussion on sites like StackOverflow
The all-topics-open view "looked much more daunting than either of the views which grouped the comments into topics"

Foefoe

Foefoe

28-yr-old female, "advanced" online experience, has looked at talk pages on Wikipedia but finds them too overwhelming/confusing to use

Test notes

When asked how she would get an overview of all the topics on this page, easily finds the 3 buttons that toggle the view of the board (chooses the middle view first, then goes to the small view).
In the smallest view, says she assumes that she can click on any topic to open it up and read the comments.
Super confused by the participant count in the smallest view. She thinks it's the number of responses, but when she investigates, she knows that's not right.
She also is confused by the "1 comment" string – not sure if it's a call to action to comment or metadata about the discussion.
Has no trouble finding the topic action menu to edit titles.
Likes the icons, says they're straightforward and let her know exactly what everything is for
Likes the way adding new topic works - works like everything else on the page.
Confused that she has IP address instead of username – wants a way to add her username to the post

Question answers

"I have looked at the discussion/talk pages. I feel as though this one seems a little more organized. I often check talk pages for more detail and to see what folks are discussing about the topic at hand, but I rarely reply because the pages tend to be overwhelming with the number of comments and responses, and I feel my own topic/comment would simply get lost in the shuffle as well. This was a lot easier to keep organized, simplified and the comment process was easy."
"I do look at discussion/comment sections on other sites. It is quite similar to what I see on blogging platforms, except that it's on a separate page rather than at the bottom of the post, or in this case, wiki article. The only real difference was the lack of ability to use your own name or handle, and no email address was required to respond (which actually made it easier)."
The topic participant number confused her the most.

Test video not available (This user declined to share his video, but notes from the test are included below)

49-yr-old male, "advanced" online experience, has looked at talk pages before and has done a little bit of Wikipedia editing "but not much"

Test notes

When asked how he would get an overview of all the topics on this page, says he wanted to see some kind of "collapse/expand all" functionality (heh), but he didn't see it right away.
He does very quickly find the 3 icons for doing that, but he claims they're pretty subtle and he wouldn't have found them if he wasn't looking for them.
Wanted to see a collapse/expand carat to the left of the topic title or a plaintext link that says "expand all/collapse all"
Not sure why there are 2 smaller views – he likes the middle the best and says he would use that one.
Confused by the fact that the comment numbers don't match between middle and smallest view.
Has no trouble clicking topics open/closed for the rest of the test.
Tries to double-click topic title to edit it. Has a lot of trouble finding the topic action menu icon, though he does eventually. He wants a pencil icon next to the tile.
He gets to a diff page by clicking on one of the last-modified pencil icons and gets very confused.

Question answers

Has seen talk pages before but doesn't recall what's different about this experience except the different viewing options.
Looks at other comments sections but thinks Wikipedia has the best.
Suggests adding text to icons to make it explicit what they do.

Talk page user tests

Test: User messaging 1: Talk page basics

This test was run on enwp so that testers would be able to interact with a full environment. Accounts were set up for the interaction, with talkpages and contributions populated for the test.

A second version of this test was run using a different article (Dwarf cat) and some simplified wording of the tasks, but the gist is the same.

Introduction

This is a test of Wikipedia's user-to-user communication systems. In this scenario, you have recently joined Wikipedia and tried a few first edits. A few days have passed since your initial login and you are now returning to the site, curious if anyone has noticed or objected to your edits.

Remember, we're testing the interface, not you. If you're having difficulty with something, the problem is with our design. Please "think out loud" as much as possible; tell us your thought process during each task, and try to explain your general opinions as you arrive at them.

If a task takes more than five minutes or so, just move on.

Tasks

Start point: https://en.wikipedia.org/wiki/Special:UserLogin

Log in using the account Silver waffles. Suppose this is the account you previously created.
See if you can find out if you have new messages regarding your prior edits, including to an article you recall being about redemption. Where would you expect this to be found?
Spend no more than a few minutes on this.
You should have found your way to this page: http://en.wikipedia.org/wiki/User_talk:Silver_waffles - If not, go there now.
What is your impression of the messages? Do they appear to have been left by a human or automatically?
Now see if you can reply to the second message regarding the article you previously edited. Pretend you disagree with it and say so (you can use a reason if you want, but it's not required).
Suppose you're not sure Orchaen solns (the user who left the message) will know that you replied. Does there appear to be anything further you can/would need to do to make sure they get your response?
Try to do this in whatever way makes the most sense to you.

Questions

Did you edit Wikipedia before this test, even once?
What frustrated you the most? What improvements would have made the process easier?
What did you like about the process, if anything?
Is it okay if we share this video publicly with other Wikipedia contributors? We'd love to show them what new members think.

Results

Aeldar02

User expected a "messages" section in user tools area.
User wishes actual messages were called out; does not understand that "Welcome Template" is also a message (there's no distinctions)
User has difficulty figuring out how to reply to messages on their own talk page
User has extreme difficulty figuring out how to leave a new message for another user
User assumes that "watch this page" will send them a "read receipt". User wants to know that their message has been sent. She somehow ends up on page history trying to see if her message was sent, and is completely overwhelmed.

Online Bee

User has difficulty understanding how to reply to messages (not sure where to reply and does not understand that original messages and replies may not be in the same place)
User actually signs a post, but does so with cut and paste - and still types her name, not knowing that the tilde codes expand
User isn't sure that message was sent, nor where it went.
User reopens message section; sees wikitext; assumes it is to edit original message and not respond (totally does not get colon indentation)

Paul

Does not understand yellow message banner. Takes a long time to figure this out.
Does not understand where messages begin and end
Accidentally clicks on "last changes" and is sent to Diff-land. Whupz.
Confused by a lack of a "reply" button
When he finally gets in, he puts the response above the section as if he were replying to an email.
Cuts and pastes tildes for response (this appears to be a keyboard problem)
Gets completely confused; adds a {{helpme}} template, which triggers a captcha.

Test Video not available (This user declined permission to share the video, but takeaways are included below)

Successfully locates new messages.
Gets lost by the edit screen
Assumes welcome message and the revert warning as a single message.
User does not respond on her talk page; instead goes to the other user's talk page.
Tries to leave a message there; gets confused. Does not see a way to reply.
User expresses confusion about the difference between responding to a section and starting a new section
User starts new section in response to original message, does not include a summary. Upon save, this results in confusion (she cannot find the message she left). Assumes there must be a delay between saving and delivery.
User tries again, this time adding a subject (thinking that the missing subject may be why it didn't appear in the first place). She correctly inserts her signature. However, this is because she see charinsert tools (the whole editor does not fit on her screen).
User ultimately gives up in frustration (this is probably why she did not give permission to share the video).

This test was the same as the first except for a minor clarification in one of the tasks (referring to a specific edit to "Dwarf cat" over generically described "edits").

GlobalE

User assumes that there will be a "messages" link in the user tools section
User thinks his talk page is a "welcome" page (given the welcome template left)
User assumes talk page will also include his contributions
User does not think the welcome template is a message.
User correctly gets the edit link
User has several issues with the editor and its usability
There is a Kafka-esque journey into existential torment as the user believes that he must cut and past wikicode for an icon in front of his message for it to be seen as a message.
User responds at the top of the message (like an email), and the section subject becomes part of his signature
User thinks a namecheck/mentioning system would be great ("It would be great if it worked like twitter where you can add an @ sign")

So he tries that, cutting and pasting their username wikicode.

Test: User messaging 2: More complex talk page usage

This test will also be run on enwp, using the usual assortment of templates and gadgets, so that testers would be able to interact with a full environment. Some accounts from the previous test were reused, and two new ones set up for the interaction, with talkpages and contributions populated for the test.

Introduction

This is a test of Wikipedia's discussion system. In this scenario, you have recently joined Wikipedia and edited a few pages, including to the Dwarf cat article. Trying to add an image, however, you found the process too confusing and had to ask another user, Orcaen solns, for help. A few days have now passed since you last logged in, and by now you expect there will have been a response to your query.

Remember, we're testing the interface, not you. If you're having difficulty with something, the problem is with our design. Please "think out loud" as much as possible; tell us your thought process during each task, and try to explain your general opinions as you arrive at them.

If a task takes more than 5-10 minutes, just move on.

Tasks

Start point: https://en.wikipedia.org/wiki/Special:UserLogin

Log in using the account 'Gleaming muffins' using the password '...'. This was the username you used to make your edits and ask for help.
See if you can find out if Orcaen solns, the user you went to for help, responded and answered your question. Where would you expect to find the response?
Can you find where you would have originally sent the message with the question? Note that in the scenario it has been a few days and you would not necessarily remember this anyway.
Spend no more than a few minutes on this task.
You should have found your way to this page: https://en.wikipedia.org/wiki/User_talk:Orcaen_solns - if not, go there now.
What appears to be the significance of this page?
Find the section 'I want to upload a picture of my cat' at the bottom. What seems to be happening here?
Can you tell who all is involved in the discussion that started from your question? What is your impression of these users?
Assume Orcaen solns' advice was helpful. How would you go about thanking him? Try to do this as a response to his explanation.
Can you tell how the messages are threaded? Regardless of what you did in the previous step, consider how you would need to thread your own responses.
Keeping this in mind, see if you can also reply to Doranton's question about your cat, keeping threading intact - suppose its name really is Binky and say so.
Now suppose you want to be sure Doranton is informed of your reply. You may have seen a blue 'talkback' message on your own talkpage (https://en.wikipedia.org/wiki/User_talk:Gleaming_muffins) from when Orcaen solns replied to your initial message, and you would like to do the same for Doranton. How would you go about doing this, or finding out how to do it?
Try to go about this in whatever way makes sense to you.

Questions

Did you edit Wikipedia or another wiki before this test, even once? If so, which?
What frustrated you the most? What improvements would have made the process easier?
What did you like about the process, if anything?
Is it okay if we share this video publicly with other Wikipedia contributors? We'd love to show them what new members think.

Results

Kayaker

User is confused by "new messages" and "last change" links in the orange bar
User thinks Welcome template is automated
User actually understands that displayed indentation means a threaded conversation
A major issues is that this user does not understand that the talk page contains several converations
User encounters the phrase "talk page stalker"; thinks its funny, but uses that knowledge to understand why a third party would get involved (suboptimal, but interesting)
User expects a "reply" link in signatures but eventually clicks edit on the section after some confusion.
POW. User is completely confused by a giant block of wikitext. Has many minor hiccups with the editor.
"I don't know where to begin." Completely confused by colon indentation; assumes that you need to know web programming; thinks wikitext is HTML. Is afraid that she'll screw up the entire page if she types anything.
"This is very intimidating to me"
Unsure which bits are manual and automatically inserted (e.g., dates, etc.)
User makes connection between colons and threading, but cuts and pastes them rather than types them.
User has significant negative experience with wikitext blocks. Expresses extreme frustration and dislike.
User does not understand that someone else would need to be "poked" to get a reply but does have a glimmer of understanding regarding "watch this page"
User is suprised that they may have actually succeeded.

kawzx7

The user does not understand signatures nor intiuitively understand indentation
User figures out signature code after some difficulty
Does not understand talkback templates are left by hand; assumes that the software will do it for him.
Confused that talkback templates are not auto-updated with newer timestamps.
Totally surprised that he can edit someone else's talk page (even though he did it earlier). Does not think he should or that he should even have the ability to do so.
User leaves reply under the wrong location (does not get talk back templates at all)
User deletes the talkback template

This test is the same as the previous, only fixing a small bug in the text of the test.

PJensen66

User attempts to use search box to find their messages; fails. Eventually finds new messages bar. Feels stupid for not seeing it earlier.
User is obviously confused by the talk page and how to use it. Ends up in several different locations (same user talk page twice, then a user page, then their own talk page, then back to the first user talk page). Discovers a flaw with user pages in general (no "leave a message" link by default).
"I don't know how to put a response in here."
User drops in and out of editing the section several times, confused by wikitext.
User gives up on responding to the messages. Is forced to move forward with the task.
User cannot find the question they are responding to within the wikitext block.
User realizes that they didn't get a signature on their response.
User cannot understand talkback templates at all and ends up on help pages trying to figure them out.
"At this point I'd pretty much give up on Wikipedia and I wouldn't bother."
User spends a great deal of time reading help pages about Talkback templates.
"I'm just kind of feeling like an idiot now."
User gives up. Declares that he would not waste his time trying to figure the system out.