Wikimedia Discovery/Search

This page describes the Wikimedia Foundation's Discovery department activities surrounding our sites' search functionality. Our current project is to improve our Elasticsearch search system (using an extension called CirrusSearch).

Search Search infrastructure improvements and maintenance Group: Discovery Start: 2015-04-21 End: 2017-12-01 Team members: Erik Bernhardson, David Causse, Trey Jones, Guillaume Lederrey, Stas Malyshev, Mikhail Popov, and Chelsy Xie Backlog: #discovery-search-sprint Lead: Deb Tankersley (product) and Erika Bjune (engineering) Updates: Status updates

See Help:Searching for a basic user guide, Help:CirrusSearch for an FAQ and a list of key features of the software component.

Current work

See Discovery's current quarter goals

Wikimedia search has an incremental search completion suggester that is in active development. The team has improved this feature with better handling of typos and better ranking of possible matches in the suggested articles. Work continues to add real-time index updates to the completion suggester and improve overall stability and performance of search by upgrading to the latest version of Elasticsearch.

This feature was available from December 2015 to March 2016 as a beta feature.

Rationale

The Wikimedia search is an important component to the discovery of knowlege within our projects....

Goals for our current quarter:

• Enhance search results and expose users to other interesting content by improving interwiki search integration.
• Adding real-time index updates to completion suggester and improve overall stability and performance of search by upgrading to Elasticsearch 2.x.

Search metrics

Key performance indicators of how search is working can be found on the search metrics dashboard at http://discovery.wmflabs.org/metrics/

Documents

• TextCat—a software component used for doing language detection
• How Elasticsearch breaks Part 1 Part 2
• 2015 Notes on unbreaking and optimizing elasticsearch
• Testing Search—testing search changes is complicated!

See also

• Phabricator workboard showing current work of the team.
• Search metrics on the Discovery dashboards
• Elasticsearch stats on Grafana
• Help:CirrusSearch - Information on how the Wikimedia search works.
  • Completion Suggester - incremental search
• Cross-wiki Search Result Improvements
• BrowserBot - a browser test bot for search
• Top Unsuccessful Search Queries - The difficulties in creating a list of unsuccessful search results.