Wikimedia Research/Guerilla Testing for iOS App

Guerrilla Testing for iOS App

In order to get some quick feedback on the edit work flow, we decided to go out into the city and find a few people to test with. We approached people in pairs, one person facilitating and the other taking notes and observing. After several observations, the group of us came together to debrief and refine notes. Then, we went back to find a few more people to test with. In the end we spoke with 7 people: 4 men (around 20-30 years old) and 3 women (2 in their 20s-30s one in her 40-50s)]

Where

We went to a coffee shop first, but it was too noisy and people were busy meeting and eating lunch. So, we went to Yerba Buena park and found 6 people willing to spend a few minutes with us running through the edit flow for the iOS App.

Tasks Tested

Making an edit using Visual Editor. Specifically, I tested making text edits and not inserting anything, adding cites, links or tables.

Observations: Hypothesis to be further tested

1. Some people are not seeing the "how did you improve the page" call to action. (5 of 7)

2. the navigation through the edit flow (and the icons: arrow and x) are not intuitive (3 of 7)

3. Specifically, what the arrow will do is not clear (3 of 7) 4. Though, to some, the edit flow is intuitive. (3 of 7)

Raw Notes

Kaity's notes

Man (20s)

Couldn't figure out how to edit section at the bottom
Expected to highlight text and be able to edit right there.
Does this go live right away? Or does someone have to check it first?
I want to see a change log and see why things were changed.
(Showed history)
This isn't very interesting, I can't see why they were changed, doesn't mean anything to me

Man (20s)

Found toc quickly
Explored menu
Scroll seems slow

Woman (20s)

Scrolled to navigate
Guessed it was a menu
I like that toc scrolls article
Finds edit right away
Finds next right away
How do you get back?
I don't know what this arrow means

Woman (50 or so)

Scrolled to navigate
I know about all these!
Menu -info about the topic
I read wiki on my iPad or scrolls easier
Don't know how to edit
Pencil could be a highlighter
Tries to hit x at first
What's that arrow for, how do you save?
Confused after preview
Tell us what you changed she found
Selected fix a typo

Maryana's notes:

Marketing app lady (20s-30s):

Didn't know you can edit Wikipedia
Doesn't want to edit -- when asked how she would fix the typo, says "I wouldn't"
But finds edit button quickly when prompted, so def. gets the iconography
Goes through proceed to preview & proceed to save pretty quickly -- finds those top right icons easily & naturally
Doesn't see "how did you improve the page"

Young kid (high school/college age?):

Knows you can edit
Finds edit button & dives right into editing without any hesitation
Finds proceed to preview but doesn't see that there's another step from there -- hesitates & asks if done
Finds save & IDs the two options right away "login & save and save anonymously", saves anon
Doesn't see "how did you improve the page"

Developer guy (20s)

Knows you can edit
Recognizes pencil as edit button but wants some fancy gesture thing to activate editing (savvy tech user, clearly)
Says he'd be scared if he didn't know wikimarkup, which he does
Gets proceed to preview & proceed to save buttons very easily
Confused on the preview screen that he's taken to the top of the section instead of to his edit -- same after he saves
Wants clearer indication that he saved something
Doesn't see "how did you improve the page"