Cross-wiki Search Result Improvements/Testing

The Discovery department is developing the ability to provide search results from other Wikimedia projects within the same language (commonly called cross-wiki or inter-wiki searches).

For example, if you were on the French Wikivoyage and searched for milk you would be presented with the expected matching articles from the French Wikipedia and results from other French Wikimedia projects (i.e.: fr.wiktionary and fr.wikiquote).

This page is an attempt to outline upcoming A/B tests that the search team would like to run and, as always, we invite all contributors to openly discuss these tests on the talk page.

After a conversation with the community, we decided to run tests on Persian, Catalan, Italian and Polish Wikipedias: testing began in mid February 2017.

Update (9Mar2017): due to a higher zero results rate than expected on the originally selected wikis, we'll expand our testing to include French, German, Russian, and Arabic wikipedias for testing.

A/B test: Add cross-wiki search results in a right hand sidebar

This is a screen capture sample depicting what a recent A/B test looked like - for displaying sister project search results

Tracked in Phabricator
Task T149806
Resolved

Tracked in Phabricator
Task T160004
Resolved

Rationale

"Why this test?" We want to find out if adding relevant search results from other wiki's (or projects) would help users discovery additional relevant information and data for the query that they typed in.

Description

"What is this test?" This A/B test will have a test group that will be shown a new right sidebar that will contain relevant search results that were found from cross-wiki searching. A control group will see the currently existing search results page.

Note: not all search queries will return snippet results from the sister projects; some of the projects might display or all of them might display, depending on the search query that was entered.

Test group users will see:

• additional search results from sister wikis in a right sidebar
• each result for the sister wiki(s) will display:
  • the top ranked result from any wiki that contain relative search results
  • an icon that denotes which wiki the result is from
  • article name of the search result
  • description of the search result
  • typical bolding of the search result term(s)
• link below the search result that is labeled 'more results'
  • this link will open a new browser tab and display a search results page for the original search term on that sister project
• separate section for multimedia results above the other sister wiki results
  • up to 3 images will be displayed that are relevant to the original search term
  • display a link that will open a new browser tab and display a search results page of multimedia for the original search term from the native wikipedia that the user is on
    • for example, if a user searched for 'rainbow' on English Wikipedia, and clicked on the more multimedia link, the user will be displayed search results for multimedia for 'rainbow' on English Wikipedia in a new tab.

Timeline

This test is expected to start February 9, 2017 and run for a week.

• Click this link to see a sample of what the test looked like - note that it is in German.

The second running of this test ran from March 16 - 27, 2017.

Tracked in Phabricator
Task T162276

Results

The test ran from February 9 - 22, 2017 and the data is being analyzed in T156300. The first draft report show that due to a couple UI bugs and a high zero results returned on the original 4 wikipedias, we had a lack of conclusive results. Therefore, we'll re-run this test; follow ticket T160004 to view the new test criteria. Namely, we'll add in four more wikis to test on and fix the UI bugs we had earlier.

The final analysis of the second A/B test can be found on Commons.

Note: There is a request for comment (RfC) that was launched by a community member on April 14, 2017 for English Wikipedia, in response to an earlier post on April 6, 2017 of the proposed upcoming deployment to all Wikipedias.

Note: As the RfC has now concluded, we'll launch the sister project snippets on all Wikipedias during the last week of May 2017, but with display exceptions to the enwiki display.

Archive

TBD

A/B test: Add 'explore similar' pages and categories for search results

This is a draft mock of a new A/B test for cross-wiki (project) search results that adds a link to "explore similar"

Tracked in Phabricator
Task T149809

Rationale

"Why this test?" We want to find out if adding related articles and categories provides users additional rich data that is more useful than what we currently have.

Description

"What is this test?" This A/B test will have a test group that will be shown additional information that will be relevant to individual results on the existing search results page. This additional information will be links and metadata of related pages and links to related categories that are similar to the individual search results. A control group will see the currently existing search results page.

Test group users will see for each search result returned:

• 'explore similar' link and dropdown icon (directional interactive arrow)
• when the user hovers or clicks on the link and/or icon, we will roll down the following display below the result:
• a label 'related pages' and
  • up to 3 related page links containing metadata (as is available):
    • thumbnail image
    • name of related page
    • metadata description
• a label 'categories' (as is available):
  • a 3 column listing of links to categories that the article was tagged with
    • up to 3 links of categories per column (for a maximum total display of 9 related categories)
• suggested languages:
  • this section shows other languages that the article has been translated into
    • this uses browser preferred language settings and the regional languages, based on the users IP.
• when the user clicks again on the link and/or icon for 'explore similar', the related pages and categories will roll back up and not be displayed

Timeline

• User testing was done usertesting.com participants on June 20 - 21, 2017
• The A/B test was started June 29, 2017 but due to a minor bug that was fixed on July 12, 2017; the official testing period was from July 12 - July 20, 2017.
• A second A/B test to display only the language links wrapped up on Sep 21, 2017, after running for a week. Unfortunately, there was only one clickthrough to an article that was written in another language, as the report documented.

Results

We will not be going forward with this feature, as noted in this ticket.

However, if a user wants to have the language link script added to their logged-in account, please follow the instructions here: https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/explore_similar#Explore_similar_-_for_languages_in_search_results.

The full explore similar script (displays related articles, categories and language links) can also be enabled for logged-in users, see this link for instructions: https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/self-guided_testing#Explore_similar

Draft images

• 
  2nd draft: search result hovered
• 
  2nd draft: similar pages link hovered
• 
  2nd draft: categories link hovered
• 
  2nd draft: languages link hovered

A/B test: Add thumbnail icons to search results

This is a draft mock of a new A/B test for cross-wiki (project) search results that adds thumbnail icons to the search results

Tracked in Phabricator
Task T149811

Rationale

"Why this test?" We want to find out if adding thumbnail images to the results provides users additional rich data that is more useful than what we currently have.

Description

"What is this test?" This A/B test will have a test group that will be shown thumbnails for the search query results and updated metadata for each search result. A control group will see the currently existing search results page. The images used in this test will be displayed using the PageImages API extension, similar to what is being used on the Wikipedia portal page in the search box type ahead feature.

Test group users will see for the search results returned:

• thumbnail (as available) image for the search result

Test group users will see for all search results returned:

• normal search results page

Timeline

We will not be going forward with this test, as other priorities have surfaced; as noted in this ticket.

However, if a user wants to add this into their logged-in account, simply add the following script to their common.js file:

importScript( 'User:JDrewniak_(WMF)/searchThumbs.js' );

A/B test: Add Wiktionary widget to search results page

Sample screenshot of Wiktionary widget with rainbow definition

Tracked in Phabricator
Task T160434

Rationale

"Why this test?" : The Wiktionary widget is a cool little experimental API that pulls information from English Wiktionary and will display a variety of things: pronunciation, a short definition, what part of a sentence it is (noun, adjective, verb) and samples of usage. Depending on the word, you might receive all of that information, some of it, or nothing at all - it all depends on what is in Wiktionary for that word.

More information can be found on the self-guided testing page.

Description

"What is this test?" This A/B test will have a test group that will be shown the widget on the search results page (if applicable); a control group will see the currently existing search results page.

Test group users will see for the search results returned:

• Wiktionary widget - if the search query has a match in Wiktionary.

Timeline

We will not be going forward with this test, as other priorities have surfaced; as noted in this ticket.

However, users can add the script into their logged-in account by following the instructions here: https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/self-guided_testing#Wiktionary_widget.

Analysis

• Discovery-Search-Test-CrosswikiSidebar
• Second_Test_Of_Cross-wiki_Search_-_Helping_More_Users_Discover_Content_On_Wikipedia_Sister_Projects.pdf