Reading/Web/Desktop Improvements/Repository/Content Separation User Testing
The Foundation's Design Strategy team conducted a survey-based experiment on usertesting.com. The goal was to measure what effect—if any—the proposed Zebra #9 update to the Vector 2022 design had on participants' reading experience.
A total of 219 participants were asked to "read some things on Wikipedia and answer some questions" about their experience. Participants were exposed to the articles for San Salvador and Guatemala City in random order. They were placed into four groups depending on whether they saw the Guatemala City or San Salvador article in the original (white) Vector 2022 look or in the Zebra #9 version of the skin.
In most cases, the page design that participants saw did not significantly affect their reading experience. Most participants (89%) reported that they did not notice the design difference while they were completing survey tasks. In spite of this, nearly half of participants (48%) expressed a preference for Zebra #9 over the original Vector 2022 at the end of the survey.
Key takeaways
- The choice of the Zebra #9 version vs the original (white) Vector 2022 did not affect participants' ratings of both articles. Participants rated San Salvador higher than Guatemala City regardless of design and order of presentation.
- Participants who saw one article in the original Vector 2022 provided similar ratings to participants who saw the same article in the Zebra #9 version.
- Most participants (89%) reported that they hadn't noticed the design difference when they were reading the articles.
- However, when participants were shown a direct side-by-side comparison image of the two designs at the end of the survey, nearly half of them preferred Zebra #9.
- 48% prefer the Zebra #9 version, 29% prefer the original Vector 2022, and the remainder (23%) express no preference.
- Surveyed participants indicated no difference in overall usability between the two different versions
Survey format
Experimental groups
Participants were split into four groups depending on which cities they saw in which designs. Each group saw either San Salvador in the Zebra #9 version and Guatemala City in the original Vector 2022, or vice versa. Each group also saw either San Salvador first in the instrument, or saw Guatemala city first.
| Group | Order of cities and designs in survey | Number of participants |
|---|---|---|
| Guatemala City Zebra #9 | Guatemala City (the Zebra #9 version) + San Salvador (the original Vector 2022) | 50 |
| Guatemala City Vector 2022 original | Guatemala City (the original Vector 2022) + San Salvador (the Zebra #9 version) | 50 |
| San Salvador Zebra #9 | San Salvador (the Zebra #9 version) + Guatemala City (the original Vector 2022) | 59 |
| San Salvador Vector 2022 original | San Salvador (the original Vector 2022) + Guatemala City (the Zebra #9 version) | 60 |
| 219 total participants | ||
The usertesting.com participant filtering system was used to ensure that participants in each individual test had not participated in previous versions of the test. (There were four survey versions spread across 22 individual usertesting.com tests with 10 participants each.) The 219 collected responses were further verified as unique by checking that each participant's usertesting.com username only appeared once in the final dataset.
Source articles
Guatemala City and San Salvador were chosen as source articles because they:
- are similar in terms of topic and content;
- have introduction sections of a similar length (~250 words); and
- have "Transportation" sections that include information about buses.
Survey tasks
Participants were asked to:
- Provide demographic and Wikipedia-use information;
- Skim introduction section of the first city;
- Skim transportation section and look specifically for information about buses;
- Repeat the tasks for the second city;
- Indicate if they had noticed the design difference; and
- Express a preference for the original Vector 2022 or the Zebra #9 version.
After skimming the introduction and transportation sections of each city, participants were posed the following questions:
- How easy was it to find information about TRANSPORTATION in this article? (Ease of finding transportation information)
- When you were reading the article, how easy did you find it to focus on the information? (Ease of focusing on information)
- When you were reading the article, how easy was it to move from section to section? (Ease of navigating)
- When you were reading the article, how do you rate the overall reading experience? (Overall reading experience)
- Express a design preference after seeing a side-by-side comparison at the end
Findings
Participants rate San Salvador higher regardless of design version
For all reading experience questions, San Salvador was rated higher than Guatemala City, regardless of whether it had been presented in the original Vector 2022 or in the Zebra #9 version. Additionally, the differences in ratings provided by participants who saw a city in the original version were not significantly different—with two exceptions—from the ratings provided by participants who saw the same city in the new design.
| city presented in Vector 2022: | San Salvador | Guatemala City | ||
| city | San Salvador | Guatemala City | San Salvador | Guatemala City |
| Finding transportation info | 4.59 (.73) | 4.34 (1.02) | 4.68 (.57) | 4.52 (.84) |
| Focusing on information | 4.27 (.86) | 3.88 (1.04) | 4.37 (.87) | 4.14 (.75) |
| Moving between sections | 4.75 (.53) | 4.53 (.85) | 4.75 (.60) | 4.72 (.64) |
| Overall reading experience | 4.29 (.71) | 3.92 (.84) | 4.34 (.71) | 4.16 (.81) |
In two cases, the original Vector 2022 design was associated with higher ratings than the Zebra #9 version for Guatemala City. Participants who saw Guatemala City rendered in the original Vector 2022 rated it higher in terms of focusing on information and overall experience than participants who saw it in the Zebra #9 version.
Participants express a preference for the San Salvador reading experience, regardless of design version
After completing survey tasks, participants were asked to express a preference for a particular article on the basis of overall reading experience: Comparing your reading experiences in Guatemala City vs San Salvador, which did you prefer? 138 participants (63%) expressed a preference for San Salvador, 55 (25%) preferred Guatemala City, and 26 (12%) expressed no preference. The design in which participants saw these cities was not associated with significant variance in these ratings.
The vast majority of participants did not notice that the two articles had different visual designs.
At the end of the survey, and after participants had already expressed a preference about the overall reading experience in both articles, they were shown an image that directly compared the two designs side-by-side. Did you notice that the Guatemala City and the San Salvador articles had different color backgrounds in their page design? Only 24 of 219 participants (11%) reported that they had noticed the design difference prior to this step.
But, when the design difference is shown in a side-by-side comparison image, participants slightly prefer the Zebra #9 version
When shown the difference between the two designs at the end of the survey, the largest group of participants (48%) express a preference for the new the Zebra #9 version.
| Which page design do you prefer? | |
|---|---|
| the Zebra #9 version (gray) | 105 (48%) |
| the original Vector 2022 (white) | 64 (29%) |
| no preference | 50 (23%) |
Discussion
Preference for San Salvador is likely due to article content rather than design choice
The choice of the original Vector 2022 vs the Zebra #9 version in this study did not affect participants' article preference. San Salvador was consistently rated higher than Guatemala City for all groups. This is likely due to a few factors:
- The design difference is extremely subtle and was not noticed by most participants;
- The survey tasks were phrased as information retrieval—participants were told to imagine that they were considering a trip to these cities, and asked to skim the introduction sections and the transportation sections of both articles;
- Participant recordings give the impression that the topics covered in the relevant sections in San Salvador were more relevant to trip planning than the topics covered in Guatemala City.