Wikimedia Research/VE on mobile

Guerrilla Testing for iOS mobile (iPad)

In order to get some quick feedback on the edit work flow on iPad, we decided to go out into the city and find a few people to test with.

The goal of this research was to observe people:

Making a simple text edit
If they can make some text bold,
If they can switch from visual editor to wiki text editor

To get started, we asked people a few basic questions first:

Do you use an Android or Apple products usually?
Do you ever use Wikipedia? (Goal of this question is to find out if they know you can edit wikipedia or not. If they answer, "Yes, I use Wikipedia", we ask "how do you use it?" If they describe that they read, mostly and don't mention editing, then we ask "Do you know that you can edit Wikipedia?"

We asked them the two getting started questions about themselves, and then presented them with an ipad that had a wiki page open (on em.m.wikipedia.beta.wmflabs.org). We had already gone in and misspelled a word, and when we presented the ipad to them, pointed out the word that was misspelled, and asked them to please correct the spelling error. Then we went through the other two questions if there was time.

Where

Monday June 2, 2014: We went to Starbucks so we could use wifi there.

Monday June 9, 2014: Internet was down at Specialty's and Starbucks, so we user-tested with WMF-ers on 6th.

Thursday, June 12, 2014: We have a Mifi to provide us with connectivity, so are free to roam the wilds of Yerba Buena Park again.

Findings: Patterns observed

June 2, 2014

All three of the participants had difficulty understanding how to save. The icons were not communicating "save" to the participants. One person who did all three tasks, was able to save successfully the second time he had to, but the first time, it took him a while to figure it out. Hypothesis to be further tested: that the icons are not clearly communicating "save", so first time users of Visual Editor might have a hard time figuring out how to save. It is learnable, so, once they figure out how to save the first time, it will probably not be as difficult.
After pressing the arrow (to save) and they were landed on the "tell us what you changed" page - they were not sure what to do next. Two people did not seem to understand that this request was to describe what they had changed, and was an intermittent step within saving. It took all three participants some thinking to understand they needed to press the green check mark to complete their save.
Requiring two taps - removing the keyboard and then clicking the green button seems to be an issue.
Counterintuitive, but both users I (Daisy) took notes on tried to click the X (cancel) to save the edit.

June 9, 2014

Hesitation on the part of the users continues regarding the general 'save edit' process - secondary save step is still mildly confusing, whether it is because of hesitancy in general or a hope on the part of users that the tester will confirm their actions (and their being able to avoid reading the content of the secondary save page)
1. Some hesitation on clicking the green save
Confusion about what the citation icon represents
Users generally apprehensive about clicking/testing out the X cancel button

Bugs and/or Suggestions

June 2, 2014

Requiring two taps - removing the keyboard and then clicking the green button seems to be an issue - not sure if this is just an iPad thing or a bug we can fix
Make the edit 'menu bar' clearer. (More) shaded background behind the icons, like for the browser Visual Editor, so users are more aware they're still in 'edit' mode
Clearer border, so the green next button is more 'in the picture'
Has been considered previously, but using text on the buttons vs symbols may be preferable

June 9, 2014

Requiring two taps issue/bug remains
Citation button dropdown menu doesn't go away upon tapping out of the menu. Will stay there unless user taps the citation button itself again
When you're in edit mode and switch editors, there should be a 'save' dialog, but didn't happen in this beta

Raw Notes

Abbey's Notes

June 2, 2014

Man (20-30s)

Getting Started Questions

He usually uses Android, uses wikipedia to read, but not to edit. He did know that it is possible to edit.

Correct Spelling Task

He went right to the edit button -
When he made the edit, he knew how to get rid of the keyboard. Once he was done making the edit, and had not pressed the arrow to save, he asked "is this final? I see the edit is made, but I don't know if it is committed." I took him time to figure out to press the arrow to save. He said it would have been easier if he had seen the word "save" somewhere.
Once he pressed the arrow, and was on the "tell us what you changed" page, he said "they want me to repeat what I am changing", and typed in the change he made. It took him a little time to find the check box that makes the edit complete.

Make something Bold

Was immediately successful with this task.
This time, he knew how to save with no hesitation.

Switch from Visual Editor to Wiki Text Editor

He said he would not edit that way, but after a few more times asking him to pretend he wanted to edit in the code, he looked around and found the switch pretty easily.

June 9, 2014

Limayli Huguet

Getting Started Questions

iPhone user. Uses wikipedia (reads, doesn't edit)

Correct Spelling Task

Found edit button relatively quickly. couldn't tell if button worked at first, pushed 'bold' then typed
Unsure what to do at save page, clicked the check box

Make something Bold

n/a

Switch from Visual Editor to Wiki Text Editor

n/a

Amy Vossbrinck

Getting Started Questions

Uses both Android and iPhone
Uses Wikipedia - reads and does a bit of editing

Correct Spelling Task

Noticed that the edit button changed/looks different
Couldn't find the save button; would want the button to say 'save'
Knew that the check button confirmed the save

Make something Bold

n/a

Switch from Visual Editor to Wiki Text Editor

Couldn't find source editing
Confused regarding visual vs source editing

Daisy's Notes

June 2, 2014

Man (30s)

Getting Started Questions

He has an iPhone
Rarely uses wikipedia. when he does, he usually scrolls to the bottom for the reference links as springboards. Doesn't edit, and didn't seem to realize he could.

Correct Spelling Task

Scrolled to the bottom of the page first, aligned to his description of his typical wiki use
Clicked top left edit button, chose visual editor - didn't know what he was choosing, and thought he could view what was edited/to be edited here
"How do I finalize?" - scrolled up/down and ultimately clicked the X (cancel) 3 separate times
Remarked that he had to invest a lot of time to edit this one word. Hit return on the keyboard, didn't finalize it either. Touched the green 'next' arrow, but that only got rid of the keyboard the first time he tried it. didn't attempt to press again. Refreshed the page, didn't work either
After prompting, got to the secondary comment page. Said he would read the Terms of Use, and clicked the check button to finalize the edit

Make something Bold

didn't attempt

Switch from Visual Editor to Wiki Text Editor

didn't attempt

Man (30-40s)

Getting Started Questions

Had two Android phones on hand
Uses wikipedia, usually gets there via web searches
'Didn't know I could' re editing

Correct Spelling Task

Clicked on the correct edit button right away. seemed to realize you had to tap once to get rid of the keyboard, but then had trouble finalizing the edit. Scrolled down, up. Tried the X (cancel) button, then tried the green next button
Clicked on the check button on the secondary save page - commented "pretty easy", "pretty simple".

Make something Bold

didn't attempt

Switch from Visual Editor to Wiki Text Editor

didn't attempt

June 9, 2014

Dennis Porter

Getting Started Questions

Android user
Mostly reads Wikipedia, has edited a couple times via edit source only, no VE

Correct Spelling Task / Make something Bold

Immediately goes to top right edit button
Taps word, uses pop-up spell check hover functionality to edit the word
Uses bold button, no issue
Upon prompting regarding finalizing the edit, immediately taps the blue next button. Hesitation upon seeing secondary save screen - 'should I have done that?'
No hesitation on finalizing with green check icon

Switch from Visual Editor to Wiki Text Editor

Part of the exploration below, not done as an official task. Clicking the edit button on the menubar was prompted, but no issue realizing what source editor was. Bug regarding switch toggle: should take to save dialog, but doesn't in this beta

Exploration

X - After going back into VE edit mode, exploring buttons: 'I think that would delete the article. But I haven't done anything on the page yet'
Link icon - 'Link'
Citation icon - 'Is that a send button?'

Dan DeJarnatt

Getting Started Questions

iPhone user, never used iPads before
Mostly reads Wikipedia, hasn't edited. Only familiar with concept of VE, hasn't used.

Correct Spelling Task / Make something Bold

Immediately goes to top right edit button
Changed a word, no problem. Highlighted a word and used the pop-up function to make it bold, instead of the button in menubar
'How do I save? I don't know'
Clicks the blue next arrow - 'Is that saving?'

Switch from Visual Editor to Wiki Text Editor

Part of the exploration below, not done as an official task. Clicking the edit button on the menubar was prompted. Confused about why there was another edit button - 'Is this maybe to draw?'

Exploration

X - cancel/delete, general apprehension about clicking this. edit mode not very ingrained here
Link icon - not sure what this was
Citation icon - thought it was for bookmarking, also possibly how to save
Katie had him try to link word snow in the paragraph to a 'snow' wiki page. Very confusing. Green check arrow already next to 'snow' in options list, confused me (Daisy) too. Not very intuitive. No way to 'save' the change. Essentially navigated via link button, navigated back, and then 'snow' was a red link. Very confusing

July 30, 2014 test

In response to previous tests, we iterated on these features

Changed X icon to back arrow, so it doesn't appear destructive
Changed arrow icon to word "next", and check icon to word "save" to provide clearer progressive actions
Changed save page to have better hierarchy and wording
Changed switch mode icon to a settings gear icon, and moved to right side
Introduced context menu for existing references and links

Findings

We tested with 4 people using same questions as before. (With addition of asking them how they would edit a link/reference)

Participants:

Used back button and it did what they expected
Hesitated when asked to save but all were able to find "Next" button
Filled out the save screen appropriately, although 1 person said it looked like an error screen at first
When asked to switch to wikitext, tapped gear icon almost immediately, but several people still struggled with "edit" and "edit source" language. Everyone also struggled with the pop-up asking them to save before switching. Might be buggy?
Everyone struggled with the context menus. Most did not notice that the icon in the toolbar was highlighted. Nobody even noticed the context bars, and didn't know what they meant when I pointed to them.

Raw notes

User 1

Uses ios, not a frequent editor

found the edit pencil easily
paused to copy content
couldn't recognize link inspector icons
would just start pressing n looking for controls
finding wikisource was easy

User 2

Has used VE before - thinks is prevents you from messing stuff up. Prefers editing in VE vs wikitext

found the edit pencil easily
knew native controls to select and bold text
link inspector visual cues are subtle
understood the word may have already been linked
said the wheel settings may have a switcher between wikitext and wikisource
was looking for something to change defaults
found wikisource the second time but said the language wasn't clear

User 3

Has used ipads before

found the edit pencil easily
was able to select and italicize easily
didn't notice the link icon highlight or the link drawer but understood that it meant that the word had been linked
successfully saved an edit
easily switched between wikitext and wikisource

User 4

Has used ve on desktop before

uses ios and android, but uses IOS more
finds pencil easily
used the native IOS dialogue for bold/itlai controls instead of the top one
says she wouldnt look at the top toolbar
didn't know what to expect once she tapped on a link
found next as an intuitive step to save
found edit source in the gear menu to switch between wikitext and ve