Guerrilla Testing Reading Behavior (App Layouts) October 3, 2014

Goal

The goal of this research was to observe people interact with the ‘stridulation’ page in the native app layout and the ‘crepuscular’ page in the prototype app layout.

How do users go about answering the prompted questions?
What do users feel are the pros and cons of each app layout?
1. How do the two layouts fare providing users with quick understanding of the topic?
Do users indicate a preference for one of the layouts and if so, why?
How well do the two layouts match up with users’ preferred styles of learning and exploring new topics/information?

Questions

To get started, we asked people a few basic questions first:

What kind of phone do you use/what OS?
Do you use a tablet, and if so, what model/OS?
Do you use a computer, and if so, what model/OS?
Do you ever use Wikipedia? (Goal of this question is to find out if they know you can edit wikipedia or not.
1. If they answer, "Yes, I use Wikipedia", ask "how do you use it?”
2. If they describe that they read, mostly and don't mention editing, then we ask "Do you know that you can edit Wikipedia?”
  1. Ask if they feel there are any specific blockers to editing.

Process and Protocol Tasks

We presented users with either an iPod Touch or an iPhone. They were then prompted to interact with the ‘stridulation’ page on the native app and also the ‘crepuscular’ page on the invision prototype.

Set-up: you’re on the native Wikipedia app homepage, with the search term stridulation in the search bar.
- (on native app) What is stridulation? From the search bar, please navigate to the stridulation page and give us a quick answer.
  - What did you learn, and show us again in app how you found the answer? (mention if ToC was activated)

Set-up: now you’ll be working with a prototype of the Wikipedia mobile app. Some features may appear or function slightly differently.
- (on app prototype) What is crepuscular? From the search bar, please navigate to the crepuscular page and give us a quick answer.
  - What did you learn, and show us again in app how you found the answer? (mention if ToC was activated; follow-up on what they expected/would like to see)
  - Could you tell me a little bit about how you felt about this experience as compared to the last version?
    - What worked, what didn’t
    - If clear preference indicated, ask for confirmation and why
  - (note down how long it took participant to find answer with native app vs app prototype.)
  - When you’re trying to learn a new task or program or explore information about a process, how do you typically learn best? (don’t suggest, just a guide! visual – seeing it done and replicating, reading – text and translate to action, other)
    - Which of these app scenarios best supports your style of learning and exploring? (meant to address lead image and caption, helpful or not)

Any final thoughts?

When/Where

Friday, October 3, 2014: We went to Yerba Buena.

Findings: Patterns Observed

7 of 8 users found the prototype version of the app more useful in viewing a representative photo of a term and being able to quickly find a definition.
1. A few users noted that the prototype layout was more user-friendly.
2. Majority of users indicated that they were primarily visual learners. The one who was a primarily text/reading learner indicated a preference for the native app layout.
Some users noted confusion at the small avatar photo in search, when it doesn’t immediately appear or is different from the first photo displayed in page.

Quotes

"I prefer prototype app because it gives a quick, concise definition right away, less scrolling, is simpler and more logical, and flows better than other version"

Bugs and/or Suggestions

Order of presentation of each layout should be at random. (DB)
The two layouts should not be specified as native app and prototype; there may be inherent bias that the prototype is better by default. (DB)
Search result photo did not always match up with the first image of the actual wikipedia page of the search term. Would be good to keep those photos the same for consistency/user mindset.

Raw Notes

Test A (ended early)

Female (15-25)
iPhone, iPad, Mac
Reads Wikipedia, usually reached from google search. has never edited, knows about it.

Task:

User glances randomly over the stridulation page, scrolls around, reads out loud the random templates, but cannot find the definition of the term. Ask-states if stridulation has to do with special animals (ended test here).

Test B

Female (15-25)
Android, no tablet, computer used (type not indicated)
Reads Wikipedia, primarily for homework and research. knows about editing (‘anyone can edit because it’s a .org’), but doesn’t

Task:

User finds and provides the definition, indicates that it’s never been a problem
User remarks that stridulation article from native app provides a better definition for the word, but prefers the prototype layout

Test C

Male (26-35)
iPhone, tablet (type not indicated), computer used (type not indicated)
Reads Wikipedia, knows about editing but hasn’t

Task:

User scrolls to text section, indicates that he wants a picture (‘didn’t take me to the top of the page’)
User feels the audio clips didn’t have anything to do with item
User likes that the prototype had a photo at the top of the page, gives him an idea of what it is. indicates that it feels like he was taken to the ‘top of the page’
- 3-5 seconds for native app, 1 sec for prototype app
Visual learner, likes pictures

Test D

Female (26-35)
iPhone/Android, tablet (type not indicated), computer used (type not indicated)
Reads Wikipedia, did not know anyone could edit

Task:

User scrolls down to text section, finds answer right away
User also scrolls down in prototype
Indicates preference for the prototype, because it is faster and has pictures and etymology
User notes that she would explore the app/program instead of getting a manual

Test E

Male (age not indicated)
Android, drawing tablet, computer used (type not indicated)
Reads Wikipedia. does not access it directly, usually through google search. ‘i have heard you can’ edit

Task:

User scrolls slowly, examining all the templates
User indicates a preference for the prototype, because the definition was right at the top
User noted that the ‘search results were too zoomed in'

Test F

Male (26-35)
iPhone, no tablet, custom-built windows computer
Reads Wikipedia, knows about editing but is too lazy to do it

Task:

User remarks that the native app has a simple interface, is straightforward and he is able to find the answer quickly
User comments that he has ADHD, and prefers the prototype app because it gives a quick, concise definition right away (slightly less scrolling), is simpler and more logical, and flows better than native app
User likes to learn basics first and tinker, and use help features on specific tasks if needed. Thinks prototype app is a better match for his learning style - it is user-friendly and more visual

Test G

Male (15-25)
Windows phone, Nexus tablet, Windows computer
More or less reads Wikipedia, doesn’t as much since leaving school. knows about editing, but doesn’t care to be involved in adding to wiki

Task:

User notes that the native app page looks the same as a web page, and that it doesn’t describe the word at first
User found the answer in the prototype app version about 90% faster, and mentions that this layout is easier to understand. also, that ‘showing an image is better’
User likes learning via Youtube video tutorials, no reading if possible

Test H

Female (26-35)
iPhone, iPad, Mac
Reads Wikipedia, knows about editing but is not interested

Task:

User wonders whether the audio icon/files at the top of the stridulation page in native app are clickable/actually functional
- Mentions before proceeding that the definition should be at the top, and there’s too much text before it
User provides the definition right away on the prototype, and states that it is much preferred to native app layout
- 90% faster in prototype
Mentions that she’s like to see more ‘fun facts’ or ‘history’ sections on pages that don’t have them

Test I

Male (26-35)
Android, iPad, Windows computer
Reads Wikipedia (mostly sports trivia), knows about editing but says he’s more interested being a user, not a contributor

Task:

User is confused at first, then scrolled and got it quickly after that. Mentions he’d like the definition on top
User gets the definition in prototype layout immediately, and likes the photo at the top
Asks to see ‘crepuscular’ in native app layout
User likes the larger image in prototype, but says it also isn’t significantly more helpful
User likes the Wiktionary infobox at the top in native app
For learning/exploring, user doesn’t like reading blog posts, but likes to use help articles from manufacturer, googling, reading and also seeing sometimes. Indicates that native app layout is better for his style of learning