Extension:GuidedTour/User testing

Test scenario one

Note: this was a test of the 'gettingstarted' tour built to support version one of onboarding new Wikipedians, where we asked English Wikipedians to try their hand at copyediting an article tagged for that issue.

Test script:

"This is a test of a new interface to help people make their first edits to Wikipedia. Imagine you've decided that, in addition to reading Wikipedia, you want to help build it. You'll sign up for a free account, and then we'll send you to a place to get started. Remember, we're testing the interface, not you. If you're having difficulty with something, the problem is with our design. Please "think out loud" as much as possible; tell us your thought process during each task, and try to explain your choices."

Create a new account on Wikipedia. (Use any name and password you prefer.)
Go to {random copyediting article with ?tour=gettingstarted}. This is a real Wikipedia article, one of a random selection for new users who are ready to try their hand at editing, but who need an idea of something to improve. We'll provide you with a basic tour of how to do so.
Follow the instructions provided to improve this page. If you get stuck, don't worry. Just explain what is confusing or where you get stuck, and feel free to move on to the next step.
When you've either made an edit (or decided that you can't figure out how), think about where would you look to find more to do on Wikipedia. Explain what you would do and why.

Test A

Did you edit Wikipedia before this test, even once?: Nope, I have never edited or signed up to Wiki before.

What frustrated you the most? What improvements would have made the process easier?: I am sorry, but pretty much everything! I really like using Wiki and was surprised at how confusing the process I was take through made it seem to edit and contribute! The sign up process was straight forward, but after I had done this, I didn't see the purpose of the page I was taken through. You need firstly to be taken to the Training section, and given a walk through step by step of what it means to contribute and edit. Then you need an interactive page that talks you through step by step what every part of the page is. You need to explain what html is, how to edit it, what it looks like in 'real preview' what process there is for checking. And after the tutorial you need to show how to find pages that need editing, or is it just that I can edit any page I like??

What did you like about the process, if anything?: Personally, as stated on q2, I love wiki and find it an interesting and informative site. If I am helping my son with his homework, we always come to Wiki to look for information. I therefore love the site, but I hated how I was then 'left in the dark' almost to find out how to edit a page! Sorry

Test B

Did you edit Wikipedia before this test, even once?: No

What frustrated you the most? What improvements would have made the process easier?: That the tool that was supposed to teach me how to edit got stuck following the first stage. From there, I didn't know how to proceed without causing more damage than good.

What did you like about the process, if anything?: It seems to have great potential, compared to the old one (which is a huge article with a list of other things to read), if it had only worked.

Conclusions

Issues identified in the tests:

Unlike users who opted to click "Getting Started" tasks, these testers had low motivation and context, which no doubt increased confusion. Tester A in particular didn't understand why she was being directed to this particular article, which is a problem obviously solved in production considering that users opt in to choosing an article from the task list.
Tester B failed to see the Preview step because it was below the fold and assumed the tour was over.
Tester A failed to see first step due to outside click
Neither tester was sufficiently prepared for the shock of wikitext, and even when they figured out how to replicate it (tester B in particular) they were clearly impeded by it.
The final step was clearly insufficient to suggest to either tester that there was value in returning to Getting Started. Testers actively looked for tasks but either didn't see or didn't understand that call to action, based on the copy provided.

User experience enhancements:

Consider an A/B test with VisualEditor, even in its current state, to measure conversion rates and how it impacts user dissatisfaction.
Design a guider after step two (click edit) and before the third step (preview) explaining to the user that they've entered edit mode and will see wikitext. Also consider opening the Help toolbar and showing it.
Force scroll the user to the tour element in a step if it's outside the viewport (i.e. fix the below the fold issue) (Trello).
Consider removing the Preview step and simply describe the optional preview in the copy of the save step.
Rewrite the GettingStarted call to action in the final step.
Disable closeOnClickOutside for the first step.

Test scenario two

In this test, we wanted to know what the impact would be if we changed from text buttons ("Next", "Okay") to the use of symbols (forward arrows, check marks). Will they make next steps less clear for users?

Test script

This is a test of a new help system for Wikipedia. In this scenario, imagine that you've decided to sign up for an account on the site in order to help expand and revise the content. This task is on a test version of Wikipedia, so don't be concerned about making correct changes to pages. Remember, it's the software we're testing, not you!

Tasks

Log in to the test site using the following credentials:
Once you're logged in visit [en.wikipedia.beta.wmflabs.org/wiki/Corgi?tour=firstedit this test page].
You should be on a test version of a Wikipedia article. What do you think you're being asked to do here? Briefly describe what you would do next and why.
If you've guessed that you're being asked to make an edit to this page, you're correct! Try to complete the task by editing the Corgi page. If you can't, don't worry, this is optional.

Questions

What frustrated you most? What would you change if you could?
What did you find most helpful about this experience?
May we share your video publicly with other Wikipedia users? This will help us show how to improve Wikipedia's editing process for new contributors.

Highlights

In the older, text-based version, users seemed to have no trouble navigating and understanding the tour for the most part.

User navigates through first two steps to edit

In the new version, users generally speaking demonstrated the same understanding. Some users failed to understand that there was a next step after being told they could edit the entire page, for which some further testing might help to show patterns. Overall, users still got most of the benefit of guiders from this version.

The user goes straight to editing, skipping the section edit step
The user understands the "edit page" vs. "section edit" dichotomy
The user saves, choosing to skip Preview
The user previews before saving

"The hint boxes/pop up boxes that specifically called out the various sections on the page that you can edit along with how to preview and save your changes."

"The pop-up boxes, they led me right to the correct menu item and also the text box that you use to edit the information was a very common looking editor that was super easy to use, just click and type. I don't think it could be much easier than that."

"The pop up bubbles highlighting the buttons to press to edit the page were very helpful"

There were some tour issues present with both versions of the tour, unrelated to the button style. For instance, users found the scrolling in the Preview step to be a distraction or confusing in some cases. "When I clicked preview, instead of showing me the content I edited as the page would look in wikepedia [sic], it scrolled down to where i was making the changes."

The user finds the scrolling down to the save area during preview confusing

Conclusions

There is no reason for the Preview step to not proceed directly to the Save step.
Showing the Preview and Save guiders before the user had the chance to complete the prerequisite step (entering text or viewing the preview) is potentially confusing or annoying for users. We should explore delaying both items, either by adding an intermediate step or waiting for the user to enter wikitext.
Several users missed the fact that there was a step after being asked to edit the entire page. This isn't bad, but we do want to educate the user about the option to edit just a section. We might consider combining the "edit page" and "edit section" steps in to one (such as by highlighting both buttons at once), or rewriting the copy to emphasize that editing the entire page is optional.

Guerrilla testing

Goal

The goal of this research was to observe people interact with the Guided Tours beta page and determine:

whether users realized that the page was a 'tutorial'
how users interact with the tour page
whether users understood the different ways to edit (full page, section-specific), how to edit, how to preview their edit, how to confirm/save their edit

Questions

To get started, we asked people a few basic questions first:

What kind of phone do you use/what OS?
Do you use a tablet, and if so, what model/OS?
Do you use a computer, and if so, what model/OS?
Do you ever use Wikipedia? (Goal of this question is to find out if they know you can edit wikipedia or not. If they answer, "Yes, I use Wikipedia", we ask "how do you use it?" If they describe that they read, mostly and don't mention editing, then we ask "Do you know that you can edit Wikipedia?"

Task

We presented users with the guided tours beta page and allowed them to interact with the tour pop-ups. Ultimately, we directed them to make an edit (add a sentence in a particular section, preview it, and save/confirm that edit).

Location

Thursday, June 26, 2014: We went to Specialty's so we could use wifi there.

Findings: patterns observed

Users tend to view the edit bubble as a call to action, not as guidance. Even when asked specifically to read and then explain what they believe to be the purpose of the bubbles, they give responses that seem to suggest that the bubbles are prompting them to edit.

Bugs and/or suggestions

From observations of 4 people in guerrilla testing and a few Usertesting.com sessions, I noticed some people pressing edit right away from the first guided tour bubble... they seem to think of it as a call to action rather than the beginning of a guided tour. Because of this, when people press edit, they miss seeing the section edit bubble. If we made the guided tour more clearly a guided tour versus a call to action, people might go through the tour first and then try to edit. Or, if we want to encourage people to press the edit button and check it out, then we design for that. Currently the guided tour has a bit of an identity crisis, and it needs to be more directly clear if we want people to just notice the edit button or press it while they are in the tour.
Users often are motivated by the first guided tour bubble to just click edit, and hence misses out the secondary tour hover.
Preview bubble appears sometimes after users save edit and return to the main corgi page.
After clicking preview, where people land on the next page seems inconsistent. Sometimes it brings users to the top of the page (where the preview can be seen), or it jumps down to where the edit box is so that the user can see the confirm/save edit hover prompt.
Users cannot click show preview button when the hover is still active, same for save button. Make show preview and save buttons clickable even when the hovers are up. (Steven confirmed that this is a bug - ACR)
One user, as he was looking at a preview view, and trying to find what he changed suggested showing Diffs in the preview. He requested this as an easier way to find what had changed. Though that might not be the best solution to needing to easily see what changed, it does bring up the issue of a need to easily see what changed. Maybe we could somehow highlight or otherwise bring a users attention to their changes as they preview.
Main issue is that it is not super clear to users that they are on a guided tour. One suggestion: Perhaps the guided tour should be more obviously a guided tour? Gray out page with just the hover bubble buttons looking 'active. Nothing else is clearly clickable until after users read and click through the two edit bubbles. After clicking through the two edit bubbles, users can click on 'edit'. Then users can explore the edit page. Perhaps everything can be grayed out after editing is done and users move toward the save button, for preview bubble to come up? No need to gray anything out for the show preview page, keep save bubble as is.)

Individual notes

Test A

Man (26-35)
Owns Android phone, iOS/Android tablets, Windows/Linux/Mac computers
Reads Wikipedia, knows about editing but doesn't

Task:

regarding the top hover, he comments that wiki can allow comprehensive, but not point edits. clicks next button
realizes that there are section-specific means of editing as well. clicks edit next to 'Appearance' section
types sentence where directed
would choose to click preview only if he'd made a 'complicated' or 'bulk' change. clearly wants to just go ahead and click save. clicks preview button upon tester prompting
he remarks that preview just brings him back to the edit window where he made the edit (this is a buggy thing. that was just a chance occurrence, and the preview brings user back to the edit window, not the actual preview part of the page)
comments that he can't really see his changes easily in preview - he wants something more distinct, wants the changes highlighted or similar
wants changes 'called out', essentially wants 'see diff' in preview mode

Test B

Woman (26-35, 36-45)
Owns iPhone, iPad, Mac computer
Reads Wikipedia, knows that people can edit, but doesn't edit. Wants to edit (esp. Arabic wikipedia), but has blockers.

Task:

reads the first bubble and excitedly clicks edit button right away
scrolls around quickly and immediately begins typing away at one of the bulleted lists on the page, getting right to it!
clicks out of the explore preview bubble
asks and reads about the edit summary, talks about how it is for the user to record notes and explain changes
clicks the minor edit box, then clicks save. very exploratory, and did not follow the guided tour.
bug - preview changes bubble appears on the corgi page after this user saved her edit. confusing
this user provided her email address for further contact on similar testing

Test C

Man (15-25)
Owns iPhone, iPad, doesn't use computers much
Reads Wikipedia, knows about editing but doesn't "haven't thought about it"

Task:

immediately starts exploring the page as he usually would, clicking on a section in the ToC
after prompting by researchers, user reviews the top bubble (edit) and clicks next to see the second bubble (section edit) as well
goes to edit menu and makes the prompted change
reviews the preview bubble, and clicks preview
appears to jump up the page after clicking preview - inconsistent, or he scrolled up quickly himself, as typically it brings the user down to the edit box so user can see the save prompt (bubble)
user attempted to click show preview button a couple times when the bubble was up, and couldn't. had to click out of bubble first
ultimately saved page and confirmed the edit

Test D

Man (15-25)
Owns iPhone, iPad, Macbook
Reads Wikipedia, knows about editing. Edits, but a long time ago.

Task:

went right to edit
interpreting first bubble as a call to action, not an 'fyi' thing or a 'guide', necessarily
user experiences inability to click on action buttons when preview bubble is up (similar to above)
after user saves edit, the preview bubble pops up on corgi page (similar to above)