Guerrilla Testing, December 17, 2014: Search and Search results

Goal

The goal of this research was to observe people interact with the search functionality in Wikipedia Alpha.

Do users understand the wikidata descriptions? How do they feel about them?
Search modes:
1. Do users notice the two search modes?
2. Do users notice when the search mode switches from ‘Titles’ to ‘Within articles’?
3. Do users understand the difference between the two modes?
4. How do users feel about having two search modes?
Can users find what they want to find?

Questions

To get started, we asked people a few basic questions first:

What kind of phone do you use/what OS?
Do you use a tablet, and if so, what model/OS?
Do you use a computer, and if so, what model/OS?
Do you ever use Wikipedia? (Goal of this question is to find out if they know you can edit wikipedia or not.
1. If they answer, "Yes, I use Wikipedia", ask "how do you use it?”
2. If they describe that they read, mostly and don't mention editing, then we ask "Do you know that you can edit Wikipedia?”
  1. Ask if they feel there are any specific blockers to editing.

Process Overview

We presented users with an Android phone. They were then prompted to interact with the search functionality, using a few search prompted queries and one of their own choosing.

Search for 'moai' (meant to test wikidata description).
Which chief justice was William Taft? (meant to prompt longer, more specific queries to test two search modes, mode switch)
Users are prompted for a topic of interest, and I ask a question that they will need to query to answer (meant to re-test search with either short or longer queries).

Protocol Tasks

Questions asked to the participant are in bold.

Search for moai, and tell me what they are – do not mention monolithic statues of Easter Island.
1. Do users understand seeing the wikidata descriptions (disambiguation)?
  1. How do they feel about some results having wikidata and others do not?
  2. How do they feel about the results generally?
Let’s say you knew that William Taft was the 28th president of the US, but not which Chief Justice he was. Can you find out using search and tell me which Chief Justice he was?
What’s a topic you’ve been interested in lately? Use search to tell me X about topic (retest within articles search).
Do users notice the two available search modes?
1. When the search mode switches to ‘within articles’ from ‘titles’, do users notice the change?
2. Do users understand what is happening and what they’re seeing?
Do users understand the difference between the two search modes?
How did users feel about having the two search modes?
Can users ultimately find what they need to find?
Overall impression and any final thoughts?
If the session went well, please invite participant to in-office concept testing.

When/Where

Wednesday, December 17, 2014: I went to Starbucks and a public space lobby on 2nd.

Findings: Patterns Observed

All users mentioned that they would simplify their queries in Wikipedia (and others additionally mention this in contrast with how they’d query on other search engines/sites). They would search for just the name of the place/person/event/etc., instead of searching in more detail. User E really wanted to test the search capabilities independent of how I was prompting (she mentioned afterward that her query on her topic of interest was not a good query for this specific application, that it was too long and detailed).
All users understand the purpose of the wikidata descriptions and find them helpful in clarification/disambiguation. One user felt that if they were present on all results, however, that aesthetics of search may be compromised (too cluttered). 3 of 5 users noticed the secondary wikidata description line (without prompting).
Search modes:
1. 4 of 5 users noticed the two search modes (at some point during the test, without prompting).
2. 2 of 5 users noticed the search mode switch on a longer query (at some point during the test, without prompting).
3. 5 of 5 users provided explanations showing their understanding of the difference between the two modes.
4. 2 users are positive about the two search modes. 1 user is very against the two search modes, calling for their removal (“useless functionalities”). 1 user likes having just one search, but likes the option to search more specifically (however, she was ambivalent about the quality of search results - felt it was mixed bag). 1 user likes having a second option (may provide more flexibility), but is overall indifferent to whether there is one or are two modes.
Users generally felt they were able to find what they needed. All users encountered some degree of initial queries not providing results or having to search for buried results on the taft search and a few on their personal queries, but the overall sentiment was positive on this front.

Bugs and/or Suggestions

5 of 5 users hit ‘go’ on the keyboard after typing in a search query, which leads to either a blank page, a ‘processing’ page with a scrolling blue ribbon up top, or (most often) a ‘page does not exist’ error page (or some combination of the three).
One user found a page result (William Burroughs) within articles search on a query, and it had a photo. However, in titles search, the same page did not have a photo.
A user used quotes in her search query string, and felt strongly that that should not cripple the app’s search capability (having quotes anywhere in her queries resulted in no search results).
A user noted that wikidata descriptions on each search result may compromise overall aesthetic of search - something to keep in mind.

Raw Notes

Test A

Female (15-25)
iPhone, no tablet, Mac computer
Reads Wikipedia, used it mainly for research at school, now looks up random stuff. Did not know anyone could edit.

Task:
Moai

User remarks on multiple moai search results. Asks me which, and I tell her moai are some type of figure. User uses wikidata to select the correct page, and mentions that she likes the search results that have the descriptors. Notices that some results have wikidata and some do not.
Overall, she likes the look of the search results. Some results are relevant to her query.

Taft

Searches for ‘william taft chief justice’, but mentions she would probably just search for his name. Scrolls through results (“this is not what I searched for”), and sees a page for William Taft very deep in the results list. Finds the answer (10th Chief Justice).
- Hits ‘go’ after entering this query. Gets ‘page does not exist’ error.

Own query

Chooses pharmacies as topic of interest. I ask her to enter a query to tell me the definition of pharmacy. She enters 'definition of pharmacy’, and in within articles finds the pharmacy page (2nd search result). She also scrolls through the other results (“I’m seeing stuff in pharmacies, types of pharmacies, links to pharmacies, Rxs…”).

Other notes

Notices the two search modes, and noticed the mode switch during the Taft search.
User thinks titles search finds article names that match the specific query and within articles search finds where the query is within the body of pages
Likes having and would use the two search mode options, not confusing / feels it is easy to use. Feels like she can find a wider variety of information with the within articles mode.

Test B

Male (15-25)
iPhone, iPad, Mac & PC computers
Reads Wikipedia and searches for whatever. Has made an edit to the page about a breed of dog. corrected some information and added a picture

Task:
Moai

User asks me to clarify after seeing moai search results, and I tell him moai are some type of figure. User uses wikidata to select the correct page. Did not notice that some results have wikidata and others don’t. Likes the descriptors, want them on other results. Would like pictures on entries missing pictures.

Taft

Mentions that usually, in wikipedia he would just search for william taft and ctrl+f to find specific relevant keywords. However, queries ‘chief justice william taft’. Ends up on a page of a high school named after Taft, not finding what he is looking for. Queries again for just ‘william taft’ and finds the information on his page.
- Also hits ‘go’ after entering a query on this task. Gets ‘page does not exist’ error.

Own query

Chooses Warcraft as topic of interest. I ask him to search for and use the results to describe a character from the game. He enters ‘warcraft blade master’, and navigates to the first result in within articles mode, which is a disambiguation page. Goes to the Warcraft 3 page (which is the redirect for blademaster (warcraft)), but is confused because the page does not directly reference blademasters (technically found what he was looking for, but doesn’t directly realize that is the case).

Other notes

User does not notice the two search modes nor the mode switch.
User thinks titles search finds pages with names specific to the query and within articles search finds query-related information in the body of pages.
The two search modes “doesn’t work for me”. Feels he didn’t find useful results in within articles results for more detailed queries.
Prefers normal search; “take out useless functionalities, remove the two modes”.

Test C

Female (26-35)
iPhone, no tablet, Mac computer
Reads Wikipedia, uses it for school research and generally searches for definitions of things. Knows about editing but prefers to consume information.

Task:
Moai

Assumes I meant the first search result. Upon prompting why and clarifying that moats are figures, she mentions noticing the wikidata description. She also says that she immediately chose the first result because it looked the most ‘legit’ - aka it is the first result, it has a nice looking photo, and it has the wikidata line. Search results can use more wikidata and pictures, otherwise good.

Taft

Mentions that generally, her queries would have more detail, but that in wikipedia she would just search for the name. Searches for ‘william taft chief justice’, spells it incorrectly and gets no results in either search mode.
- Corrects the spelling, clicks ‘go’, and gets the ‘page does not exist’ error. “seems weird”. After prompting her back to the search drop down, she finds the answer on taft’s page.

Own query

Chooses escrow as topic of interest. I ask her to search for and use results to explain something about escrow; she searches for ‘escrow timeline’. She remarks that all the results in within articles search are random, and she doesn’t get results in titles search. At this point, she says she would just search for something simple and direct ‘escrow’.

Other notes

Notices two search modes, but did not notice the switch.
User thinks titles search returns articles specifically matching the query and within articles search returns articles with the query mentioned in content.
User likes just having one search, but also likes being able to be more specific: “within articles search mode could be helpful depending on what you search for […] maybe helpful, maybe not." Feels search functionality is generally pretty straightforward.

Test D

Female (26-35)
Android, Kindle Fire, Mac & PC computers
Reads Wikipedia, uses it mainly for research at school (HCI student at Carnegie Mellon). Made a factual edit on a page back in the day.

Task:
Moai

Sees a few options. Mentions that my saying figures as a clarification allowed her to match up the result with the associated picture and description. Notices that some results have wikidata and others do not. Thinks results are more useful with wikidata.
Mentions that more photos would be helpful, otherwise some people searching for terms they are unfamiliar with won’t know which to choose.
Notices a moai disambiguation page very low in the search results, suggests that this should be higher up on the results list.

Taft

Normally, she would search for ‘william taft chief justice’. on wikipedia she would just search ‘william taft’. Prompted to try the normal search - she scrolls through many results in within articles search mode, comments that none of them seem to match, until reaching taft’s own page pretty low on the list.
- Also hits ‘go’ after entering a query on this task. Gets blank page, then a ‘page does not exist’ error.

Own query

Chooses William Burroughs as topic of interest. I ask for the date he accidentally shot his wife, and she searches ‘william burroughs murder’. Finds his page in the within articles search results, and tells me the date. Searches ‘william burroughs’ to see if the results would be different.

Other notes

User notices two search modes, but did not notice the mode switch.
User thinks titles search returns articles where the main subject is in the query and within articles search returns pages where the query subject is mentioned, but perhaps not the main topic
Likes having two search modes. When searching 'william burroughs', the search stays on within articles. She switches over to titles search, and notices that the william burroughs page (the same one where she answered my question) result does not have a photo, even though it did when she found/selected it in within articles earlier. “weird”.
User feels it was difficult to find the taft result; his page was too far down. However, she does mention that she thinks it is helpful to be able to search in context.

Test E

Female (36-45)
Android, no tablet, PC & Linux computers
Reads Wikipedia and uses it for general fact-checking; knows about editing, but has not felt the need.

Task:
Moai

Asks me which moai result she should use; ends up using my response to select the result with photo and wikidata description. Mentions that this would be helpful to people for disambiguation or for people searching for something they don’t know much about.
For the moai search, she mentions later that some search results have the wikidata, others have parenthetical clarifications, and others do not… however, she notes that she did not specifically notice the wikidata description.
Finds wikidata helpful; however, she mentions that if it were to be 3-4 words for each search result, it may look cluttered; “balance clearer search results with aesthetics”.

Taft

Enters 'william taft’ ‘chief justice’ (with quotes), no results on either mode. Removes second part of query, still with quotes; still no results. “would’ve expected to see his name as the first result here”. Searches ‘william taft chief justice’ without quotes; sees a lot of “random, perhaps related” results until she finds taft’s page.
- User also accidentally clicked on Taft III page; however, notices the disambiguation and realizes immediately that it wasn’t the right page.
- Mentions that putting quotes in shouldn’t affect search like this. She says that she usually uses quotes and booleans and other search tools in normal google-type searches - seems to me that she already felt like she simplified her way of searching for wikipedia.
- Also pressed ‘go’, and received the page does not exist error.

Own query

Decides to continue to flout her own understanding of how wiki search is supposed to work and searches for ‘natural hazards and urban resilience’. No results in titles, but within articles results has psychological resilience and then, further down, urban resilience (she seems satisfied with that, though).

Other notes

User did not notice the two search modes initially, but does after the taft search. Notices the mode switch.
User thinks titles search matches query to name of wiki articles and within articles search finds articles where the query string is mentioned.
Feels that within articles search seems to provide more flexibility. However, no particular preference for whether there is one overall search mode or two separate. Feels she was able to find what she needed.
Mentions that it’s really important for the user to understand and be familiar with any particular system to be able to query it correctly. She feels that her own query was a relatively poor query for wikipedia app (she would generally simplify a lot more, use short and specific terms).