Reading/Web/Desktop Improvements/Updates/2022-07 for the largest wikis

Defining a process for the discussion of making Vector 2022 the new default

About the project

In short: in one to two weeks, we would like to begin the conversation to update the default design of Wikipedia to Vector 2022. We are interested in deciding on the process we want for the conversation before discussing the change itself. This is because we want to have a good decision-making process that allows for a way to identify the needs of the community from the new skin.

Hi everyone,

We would love to see the Vector 2022 skin (see what it looks like) become the new default across all wikis, including [insert your language] Wikipedia. The skin would be turned on for all anonymous users, and also all logged-in users who now use Vector (the current default). Logged-in users are and will be able to switch to any of our other available skins, including the current Vector. We will be ready to begin making the change at the end of August (and not in July, as previously announced), when the visual refinements and other deployment blockers are ready.

The goal of the project is to make the interface more welcoming and comfortable for readers and useful for advanced users. The project consists of a series of feature improvements which make it easier to read and learn, navigate within the page, search, switch between languages, use page and user tools, and more. The team has been working on this change for the past 3 years, ensuring that every change is thoroughly tested and proven to work.

Every one of our changes goes through a rigorous process. First, research, then development, qualitative and quantitative testing with readers and editors, prototype testing with editors (across 30+ language communities), more changes based on the communities feedback, and monitoring. When a change does not meet the success criteria or does not perform better than the previous version, we either stop developing the change or work on it until performance is improved.

The changes we have made will be crucial to making the site more welcoming and easier to use to new readers and editors. When compared to the old Vector skin, Vector 2022 is proven to:

Save time while reading and editing (measured by decreases in scrolling to the top of the page after the introduction of sticky header and table of contents)
Increase exploration within a given page (measured by increased clicks to the table of contents)
Improve readability (measured via collapsible sidebar usage)
Improve the discovery of new content (measured by an increase in searching)

More details on how we have been building Vector 2022

Unhide to read

Problem identification research with readers and editors - during this phase, we studied the way people use the site. We identified the largest usability issues as well as issues to exploring the site further, becoming more engaged with reading or editing. We did this by interviewing readers and editors across multiple countries and locations
Prototype development and testing - this is when we build out the ideas of a feature and begin showing new things to our audiences. Each feature was tested with readers and editors through interviews and rounds of prototype testing. Generally, for testing with editors we used central notice banners across multiple language wikis so that we can get the most diverse audience possible. Each prototype was tested by approximately 200 editors on average
Refining and building - we then take the feedback from the prototype testing and refine or change the prototype based on what needs were identified in the prototype testing. In some cases, we ask for additional feedback so that we’re sure we’re making the right decisions
A/B testing on pilot (early adopter) wikis and other quantitative testing - this is the "beta" phase. We test whether the feature works as expected based on the criteria of success we have previously defined. For example, the sticky header was designed to decrease scrolling to the top of the page. We gave the sticky header to 50% of users and compared them to the other 50% for two weeks. After two weeks we compared the results and identified that that people that had the sticky header were indeed scrolling less to the top of the page in order to select any of the tools available there. If we get negative results from our test, we change the feature and test again. During this phase, we also monitor usage across all wikis, where many account holders are already using the new skin.
Finally, we deploy the feature on more wikis and continue monitoring the way people are using it so that we can flag any issues.

We are working on an easy way to explore all of the above data and research (and are welcome to suggestions on the best format). For now, the best way to see the testing is:

From Reading/Web/Desktop Improvements/Features, select the feature you are most interested in
Within that feature page, refer to the Qualitative or Quantitative testing section to see the results and our conclusions

For more details on this data, please visit our repository.

A quick overview of our learnings

Collapsible sidebar

The collapsible sidebar allows people to collapse the main menu in order to focus on reading - helping to find the information needed without distraction

Qualitative testing with readers and editors on the usefulness of the sidebar and our navigation. Our conclusion here was that the number of different tools provided on the page by default was found to be overwhelming by readers and actively discouraged them from reading, but also from exploring the functionality within the page, an effect opposite of what the exposure of multiple tools aims to do. More details can be found on our feature page for the collapsible sidebar, as well as within the original report
Quantitative testing on the usage behavior of the sidebar itself, in both its open and collapsed states (see the results). When using the sidebar, logged-out users are much more likely to collapse it and, once collapsed, to keep it collapsed. In addition, the rate of un-collapsing also indicated that users are aware that, were they to need to navigate to an item in the sidebar, that option was available to them.

Maximum line width

We have introduced a maximum line width to articles. Research has shown that limiting the width of long-form text leads to a more comfortable reading experience, and better retention of the content itself.

Our studies with readers showed that readability was an issue with the current interface, in particular being able to focus on the content
Pages that are not in a long-text format (such as diffs, special pages, page history) will be presented at full-width as before
Logged-in users who wish to read articles at full width are welcomed to set up a script or gadget that will allow for this, such as this one
For more details on research and motivation, see our research section

Search

The new search widget includes important context that makes it easier for users to find the query they are looking for by adding images and descriptions for each search results

People had difficulties finding the correct result using our previous search
Our A/B testing showed that adding the new search can lead to a 30% increase in search sessions initiated on the wikis we tested

Language switching

The new language switching tools are more prominently-placed than before. They allow multilingual readers and editors to find their preferred language more easily.

Readers did not previously know they could switch languages from the page, even if they read multiple language wikis habitually. They would use external search engines to find the correct article instead.
In our user testing, new readers were able to find the new location much quicker than the previous location
Our qualitative testing showed that this was more difficult to find for existing users who were used to the previous location, leading us to iterate on the feature. We have since added a note in the previous location of the language switcher and made the button itself a more prominent color
In the future, we will continue exploration on languages, considering potentially a direct link to a person’s most frequent languages

User menu

The new user menu provides links to all links related to the user in one place. This reduces confusion between general navigation links and specific user links

New editors were confused between the links at the top of the page and other navigation. They didn’t know these links pertained to their personal tools
Our user testing with readers and editors showed that people found it intuitive that all user links are in a single menu and that the menu is easy to find
In our prototype testing, 27 out of 38 (71%) editors and other logged-in users showed strong positive experiences with the user menu
Based on community requests and current data, we iterated on the feature and moved the watchlist link out of the user menu for easier access

Sticky header

The sticky header gives access to functionality that is used most frequently that was previously only accessible at the top of the page. The goal is for people to scroll less and thus, save time

Our A/B test showed an average 15% decrease in scrolls to the top per session for logged-in users within the 15 pilot wikis we tested on

Currently, we are finishing the predicted part of the development work. We think this is the best time to plan for any additional work that might need to be done based on needs of the various communities. We hope to begin this process during this conversation.

What should the process look like?

We need your help and feedback on how to proceed. We have two requests:

We need to talk in a way that works well for the [insert your language] Wikipedia community. What would be the best format and timeline to discuss the change? We have included a proposed format below, and are interested in what you think about it. If you agree, we can begin the deployment conversation in one week. Here is our suggestion:
1. Have the deployment conversation that would take 2 weeks. The goal for that discussion will be to identify breaking issues or opportunities for improvement for the new skin. It will be important for us to reduce the risk of bugs or imperfections that would be particularly troublesome on [insert your language] Wikipedia
2. After the deployment conversation, we get back to you with a prioritized list of remaining work/fixes necessary prior to deployment
3. Before the deployment,
  - Banners announcing the change will be displayed for logged-out and logged-in users
  - The announcement will be made both on the Village Pump as well as in the Tech News.
4. We proceed with deployment once the agreed upon fixes are ready.
We need to understand the perspectives of different parts of the [insert your language] Wikipedia community. What forms of communication would help to gather feedback and further raise awareness for the [insert your language] Wikipedia community? We would like to have an open discussion, but are open to other forms such as office hours dedicated specifically for the [insert your language] Wikipedia community, or guest presentations at community meetings. If necessary, we can also try to adjust the timeline of conversations based on your needs.

We welcome your replies here, or via email (olga@wikimedia.org, sgrabarczuk@wikimedia.org), as well as during our next office hours.

Thank you for your time and help.