Wikipedia.org Portal A/B testing

We want to ensure that the Wikipedia.org Portal page is easy to use and understand while promoting further discovery into Wikipedia and it's various sister projects.

The Portal page has around 10 million daily pageviews (as of Oct 2015) (now over 13 million daily pageviews as of May 1, 2016). This is awesome. However, a majority of our visitors simply leave the page without doing a search, selecting a language link, or clicking into a sister project page. We want to change that by presenting intriguing and interesting content with a more modern look and feel to the page.

The Discovery Department's Portal Team recently updated the search box on the wikipedia.org portal page. The search box is now more prominent and displays metadata in the dynamic type-ahead search results to help visitors choose exactly what they were looking for.

Now, it's time to refresh the page to be more dynamic and visually stimulating, so that more people can easily access the content available in the Wikimedia projects. To do this, starting in Q4 2015/2016 (April - June 2016), the Discovery Department's Portal Team will conduct a number of A/B tests and analyze the data from our visitors. Once testing and analysis is done for each test, the Portal Team will publish the results, and if the test was successful then we'll look at pushing the experiment to production.

Below is a list of upcoming A/B tests and updates that the Portal Team will tentatively perform; feedback and input is welcomed on the talk page. The mockups represent an idea of what we want to do, and may not represent the actual final test or release to production.

For more information about the Wikipedia.org Portal, please refer to this page.

A/B test: browser language detection (completed)

B control group treatment for A/B language detection test (March 2016)

Rationale

"Why this test?" The Portal Team is curious whether Portal visitors who do not have English set as their preferred browser language would be more likely to take an action (click through to a wiki site) if their preferred language link is in a more prominent and visible position.

Description

"What is this test?" In this test, the portal code will detect what the visitor's browser's preferred language(s) are and then re-sort the Top Ten language links around the globe to match those language preferences.

Wikipedia portal - localized primary links and wordmark, mobile view

Also included in this test, will be a localized language phrase positioned below the Wikipedia work mark: The Free Encyclopedia. The phrase will be based on the visitor's first preferred browser language. If there isn't a translation for the preferred language, the English phrase will be displayed. The same phrase that is currently underneath all the top 10 language links will be removed to provide more white space and a more cleaner look and feel to the page.

To view the visual treatment of this test that the B Group will see, visit this page. The A Control group will see this page, which is the current implementation of the Portal page.

Timeline

This test ran on wikipedia.org from Tuesday, March 22, 2016 - 3:00 PM UTC to Thursday, April 14, 11:02 PM UTC. This extended testing time was necessary to collect a diverse dataset of non-english speaking users (see Phabricator ticket)

Archive

A functional version of this A/B test is available to view here.

Results

An analytics report delving into the results of this A/B test has been created and is available here: https://commons.wikimedia.org/wiki/File:Wikipedia_Portal_Test_of_Language_Detection_and_Primary_Link_Resorting.pdf

A/B test: adding descriptive text to sister (other) project links (completed)

wikipedia portal footer - sister projects

Rationale

"Why this test?" Usage of the sister (other) project links is quite small in comparison to Wikipedia and the Portal Team would like to find out if adding short descriptive text to each link would create investigation into those projects by the portal visitors.

Description

"What are we trying to do?" This test will keep the page in a state consistent with any previous successful test / production release and add in a few words to each sister project link. The few words added to the project links are meant to be short but descriptive of what the visitor to those project can expect to find.

The exact wording of the descriptive text will be determined, at some time in the future, but the accompanying draft mocked image shows one way we might want to display them. This update will be to see if the analysis shows increased click-through to the sister project sites with the addition of the descriptive text.

Timeline

This test ran on wikipedia.org for a week from Tuesday May 3, 2016 - 3:00 PM UTC to Tuesday May 10, 2016 - 3:00 PM UTC. Phabricator ticket

Archive

An archived version of the test is available at the following URL:

https://people.wikimedia.org/~jdrewniak/descriptive-footer/index.html#descriptive-text-b

Results

• Analysis of the data from this A/B test is here.
• We refactored the code and released the descriptive text to production on May 18, 2016.
  

A/B test: languages by article count (completed)

Rationale

"Why this test?" The Portal team wants to find out if the listing of languages by article count could be displayed in a more streamlined way without significantly increasing the bounce rate or decreasing the clickthrough rate. Our goal is to promote easy scrolling through the long list of languages by article count, but in a dropdown format that is above the fold on the page.

Description

"What is this test?" In this test, there will be a selection dropdown added directly below the search box on the Wikipedia.org portal page. When a visitor clicks on the dropdown, they will see the familiar look and feel of the existing languages by article count treatment, with the top most languages by article listed first. By inserting the article counts into a dropdown, the page will be much simpler and streamlined with an addition of easier-on-the-eyes white space. Included in the dropdown, will be all the languages that Wikipedia supports. All the language links will be clickable, enabling the visitor to quickly go to that language specific wiki site. This test is not meant to affect the click-through rate of the language by article count links, but to put the sometimes confusing information into a easier to understand interface.

Archive

An archived version of the test is available at the following URL:

https://people.wikimedia.org/~jdrewniak/collapsed-languages/index.html

Timeline

This test was started on June 6, 2016 and ran until June 14, 2016. Phabricator ticket

• 
  languages by article count on desktop
• 
  wikipedia portal - language dropdown on desktop
• 
  Wikipedia portal - test for collapsing articles by language, mobile view before clicking on button
• 
  Wikipedia portal - test for collapsing articles by language, mobile view after clicking on button
• 
  Wikipedia Logo

Results

• Analysis of the test is here and below is the summary of the findings:

One of Discovery’s Wikipedia Portal team’s goals is to make wikipedia.org a more inviting experience. One thing that worried us, from a user experience perspective, was the large section with links to Wikipedia in all the different languages. Specifically, we were worried it might be overwhelming to a new user. To that end, we propose a design change wherein that section is made optional and redesigned to have a cleaner, columnar layout.

We ran A/B test to assess the effect of this change for a week, from 6 June 2016 through 14 June 2016. The test group consisted of 3019 users who received the cleaner design while the control group consisted of 2560 users who received the current design. We found that the test group had a 3.64% higher probability of interacting with the page and was 1.07 times more likely to engage with the page than the control group. Furthermore, although both groups received the dynamic primary link feature, users in the test group had a 6.82% higher probability (1.09 times more likely) of visiting a Wikipedia in their most preferred language.

Our main goal with this design change was to decrease the bounce rate (increase the engagement rate). We saw a significant improvement in how people engaged with the page when given the cleaner variation, which suggests that we should proceed with this change.

Conversation with the community

We posted a page here to talk about implementing this into production.

A/B test: add trending articles (upcoming: TBD)

add trending articles using previous tests

Rationale

"Why this test?" As a way to show more interesting articles to our portal visitors, the Portal Team will add trending and/or most viewed articles to the Wikipedia.org portal page.

Description

"What is this test?" This test will keep the page in a state consistent with any previous successful test / production release and add articles to the page to entice visitors to find out more information without having to specifically search for them.

The method of determining the trending (or most viewed) articles will be very similar to how the Wikipedia mobile app determines which articles to display.

Exactly how and what articles to be displayed is still to be determined, as well as if the articles displayed will be localized to the visitor's preferred language and / or location. The test will be to see if the analysis shows increases click-through by visitors to the articles that are displayed.

Timeline

Phabricator ticket

A/B test: top 10 language links (declined test)

top 10 language links

Rationale

"Why this test?" The Portal team wants to see if collapsing the top 10 language links increases the click-through rate to those specific wiki sites.

Description

"What is this test?" The top 10 language links will be collapsed into a single (or possibly two) lines of links above or below the search box. There will be an additional link added to include a dropdown selection of all language wiki links.

The language by article count will be reverted back from what we displayed in test 2, thus displaying what is currently on the Portal page (list of articles by language sorted highest to lowest amount).

In this test, we want to see if the different treatment will affect the average click-through rate on those primary language links. The top ten language links will in the order of the visitor's browser's language preference (if they have one) from left to right. If the visitor's browser does not have more than one preference, we will display the top 10 languages in order of page views.

Timeline

Phabricator ticket

A/B test: combination of above tests (declined test)

combination of tests 2 and 3

Rationale

"Why this test?" Once the Portal Team has done the testing listed above and the analysis shows measureable successful results, we'd like to take the next step of combining the successes into one page for another test. This test will be similar to the previous tests, in that we don't want to have a significant increase in the bounce rate on the Portal page or a significant decrease in the click-through rate for any of the links that will be displayed in a different manner.

To determine the success of this test, we'll be comparing it to the baseline currently established on the dashboard for the Portal page. Ultimately, this test would go into production, once analysis is done and code is finalized.

Description

"What is this test?" This test will combine several tests that at this point, the Portal Team have successfully tested. It would entail displaying the:

• preferred browser language(s) in order and localizing The Free Encyclopedia phrase,
• adding descriptive text to the sister (other) projects links,
• putting the top 10 language links into a new horizontal visual treatment, and
• showing all languages (ordered by article count) in an easy to use dropdown.

Timeline

Phabricator ticket

See also

• Testing - the Discovery team's A/B testing guidelines and approach
• Portal A/B Testing - Developer testing workflow