User:Deskana (WMF)/Proposals/Q1Analysis

Define and measure user satisfaction with search result relevance

Background

In order to make sure a team is choosing the correct work to satisfy the users of their product, understanding how those users use the product, and knowing what works well and poorly for them, is of crucial importance.

In the case of search, more traditional metrics such as conversion are not ideal as they can track false positives; if a search engine is presenting irrelevant results, users who click on a search result might have done so out of frustration because that’s all they could find rather than because the result was what they wanted. In order to truly understand whether users are finding what they need quickly, easily, and with minimal frustration, additional data must be collected.

In Q4 2014-15 (Apr - Jun 2015) the Discovery Department implemented a data collection schema for search result relevance satisfaction, and deployed an initial version to production. The team will spend Q1 2015-16 (Jul - Sep 2015) improving this schema and preforming necessary refinements to use the schema as a KPI (key performance indicator) from Q2 2015-16 (Oct - Dec 2015) onwards.

Objective

Further our understanding of whether our search is giving our users relevant results by finding a quantitative metric to measure user satisfaction with search, and using it in production to measure satisfaction with our search.

Key results

The key results that will evaluate the success of this work are:

• Define and communicate the metric that will be used to measure search satisfaction.
• Implement data collection to measure this metric in production.
• Visualise this metric on Discovery Department dashboards.
• Iterate until the metric is stable enough to be used as a KPI for the Discovery Department in Q2 2015-16 (Oct - Dec 2015).