Wikimedia Research/Guerrilla Testing Table of Contents discovery (mobile)

Guerrilla Testing for iOS app: Table of Contents discovery

In order to get some quick feedback on whether users could discover and utilize the Table of Contents icon feature on the iOS app, we decided to go out into the city and find a few people with whom to test.

Exercise: Find and read the first sentence in the 'economic policy' section of Barack Obama's Wikipedia page.

Goal of this research: determine whether users found the Table of Contents function to be discoverable and an effective tool for completing the exercise.

To get started, we asked people a few basic questions first:

1. What kind of phone do you use? Y/N. iPhone, Android, other
2. Do you use a tablet? Y/N. iPhone, Android, other
3. Do you use a computer? Y/N. Mac, PC, other
4. Do you ever use Wikipedia? (Goal of this question is to find out if they know you can edit wikipedia or not)
   1. If they answer, "Yes, I use Wikipedia", we ask "how do you use it?
   2. If they describe that they read, mostly and don't mention editing, then we ask "Do you know that you can edit Wikipedia?"
   3. If they say yes, they know about editing, but haven't, we follow-up with "What has stopped you from editing?"

Where

Tuesday June 15, 2014: We went to Specialty's. The internet there and the mifi didn't work, so we didn't use our recording device nor did we use google air to record the sessions. We also could not start on another page and have the user navigate to the Barack Obama page first.

Findings: Patterns observed

June 15, 2014

High level findings for this test is that there are issues with discoverability of the ToC via the "hamburger" icon. How might we make it more obvious that there is a ToC?

Only two of the 7 people found the ToC without any prompting. Three of the people found it after slight prompting (something like "do you see a table of contents anywhere?" or "there is a table of contents, can you find it?"). Two of the participants never found the ToC. Since the only two people who found the ToC independent of any prompting were directly told it is an app versus a mobile website, we would like to test with some more people, making sure they understand it is an app versus a mobile website and see if understanding that context is something that helped them find the ToC.

Of the 5 people who eventually found the Toc, all of them understood what the ToC was, they were easily able to interact with the ToC and understood that the text on the left was related to the ToC. There seems to be no problems interacting with the ToC once it is discovered.

1. First three - we mentioned we were testing the wikipedia app, but we didn't make it a point to frame the experience and explain this was the mobile app, there is also a mobile wiki browser and a normal wiki accessible via browser on mobile/desktop
   1. 2 found the ToC after prompting of some kind "there is a ToC", "take a further look"
   2. 1 never found it, had to be shown the button
2. Last four - we framed the experience more thoroughly and explained the different ways of accessing wiki (on mobile and otherwise), and clarified that we were on the mobile app
   1. 1 found immediately, no scrolling or hesitation
   2. 1 scrolled briefly, then found independently
   3. 1 scrolled, looked through the infobox content, then found (but indicated she wanted to "check out" what it was, so not immediate recognition of functionality)
   4. 1 scrolled, looked through infobox, was prompted, but never found

Bugs and/or Suggestions

June 15, 2014

Suggestions

1. User #2 suggested different options for the ToC icon ("the lines don't tell me much")
2. User #6 suggests putting the ToC button on the left side of the screen
3. After scrolling down, the menu bar (along with the ToC button) does not show. When scrolling up, the menu bar appears. Possibly, if it were always present as people scroll down the page, it may be more discoverable. Let's test this?

Raw Notes

Daisy's Notes

June 15, 2014

Male, Age 46-56

Getting Started Questions

1. iPhone 3
2. iPad, Surface
3. Macbooks
4. Reads wikipedia, does not edit but knows about it. "writing's hard, good writing's harder". daughter edits. called wikipedia "source of all knowledge", semi-jokingly

ToC Task

1. "This [referring to mobile in general] is the problem - being able to search"
2. scrolling down quite a bit, doesn't seem to be trying to find a ToC
3. notices and uses ToC on desktop. generally likes to use ctrl+f to search for keywords
4. thinks the infobox is the ToC. still scrolling
5. Abbey prompts finally to click the ToC button
6. finds the economic policy section. "interface is a little bit tough"

Male, Age 15-25

Getting Started Questions

1. iPhone
2. iPad
3. iMac
4. Reads wikipedia, does not edit but knows about it. Usually finds answer he's looking for - if not, still too much of a time investment to find out and make the edit

ToC Task

1. "[section] won't be near top in general bio..." mentions that typically, wikis have contents broken down into sections (talking about collapsible ToC section)
2. comments that he sees a link about economics
3. mentions also that he would normally search inside article, or if time allowed, scroll through article
4. Daisy specifies to search for the section name, then describes ToC button functionality. user initially says not familiar with the term 'table of contents' wrt wikipedia pages (technically called contents)
5. after some more prodding, user finds the button
6. "this is app as opposed to the website." said he would search within mobile page, says mobile is a nice experience (user is used to having the collapsed ToC section

Male, Age 26-35

Getting Started Questions

1. iPhone
2. iPad
3. Dell PC
4. Reads wikipedia sometimes, does not edit but knows about it. time commitment is usually the deterrent to editing

ToC Task

1. scrolls down the page, looking for the ToC but remarks that I can't find it
2. after a hint that the ToC is indeed a present feature, user finds it and remarks that it "is there" and is "much easier"
3. comments on the icon "doesn't tell me much". suggests book/chapters representation on icon instead

Beginning with this user, Abbey and I began clarifying the differences between the mobile app, the m.wiki mobile site, and the 'normal' wikipedia users can access from any browser on mobile. We felt it was possible that we didn't do sufficient pointed framing of the experience as interaction with a mobile app.

Female, Age 26-35

Getting Started Questions

1. iPhone
2. HP tablet
3. PC
4. Reads wikipedia, does not edit but knows about it. hasn't thought about editing much; says she's "lazy"

ToC Task

1. begins scrolling down, immediately but hesitantly. remarks that "usually there's a dropdown" (referring to ToC)
2. looks through the infobox, scrolls around, taps on the ToC (all relatively quickly). Abbey prompts her to discuss what she thinks it is/does - "breaking down into an outline"
3. clicks the economic policy section - "a lot easier"
4. Abbey asks how she usually finds things/navigates wiki pages - "depends" (scroll, use ToC, ctrl+f)
5. says that the "difference [with this app] is that I have to click it" (i suppose vs in desktop)
6. comments that she likes the contrasting dark background, and that scrolling through the ToC 'outline' concurrently scrolls the page content as well

Male, Age 46-56

Getting Started Questions

1. iPhone
2. iPad
3. Macbook Air
4. Reads wikipedia, does not edit but knows about it. hasn't thought about editing

ToC Task

1. scrolls down quickly and briefly through the infobox (couldn't see if he even scrolled to the non-infobox content parts), goes to the ToC immediately after this

Male, Age 26-35

Getting Started Questions

1. iPhone
2. iPad
3. Macbook Pro
4. Reads wikipedia, has edited, but not as often as he used to. commented about how people "quashed" his edits, or he wanted to make an edit or change but it is already done by the time he attempts

ToC Task

1. doesn't even scroll, goes to the ToC button right away
2. feedback: put icon on the left side. comments that it is more intuitive and more people hold their phones with their left hand and uses that hand to navigate
3. makes a note to us that google provides search results directly to the mobile wiki site

Male, Age 26-35

Getting Started Questions

1. iPhone
2. iPad
3. Macbook & PC
4. Reads wikipedia, does not edit but knows about it. hasn't ever felt strongly enough to edit (something very inaccurate, or something missing on page or subject he feels very strongly about)

ToC Task

1. scrolls down first - wants to "see how big this page is"
2. combs through infobox - comments that he "would do ctrl+f"
3. scrolls up very quickly all of a sudden - says he's looking for ToC. goes through infobox again
4. was not able to find the ToC button, even after more prompting (*note: device/internet malfunctioned further, and only pulled up the page split (black background, but no ToC content. could not demo ToC section to this user)