In Search of Recall

Precision and Recall

In information retrieval terms, this focus on the first page corresponds to an emphasis on precision, the fraction of results that are relevant. To be more precise — no pun intended — it corresponds to an emphasis on position-biased precision measures like discounted cumulative gain (DCG).

When Recall Really Matters

There are three search scenarios where recall is especially important:

  • Searches that return no results or only a few results. For these searches, even a small increase in recall can have a critical impact. When the number of results is low, the expected benefit of increasing recall tends to outweigh the expected cost of decreasing precision.
  • Searches for which the best results do not match the keywords. The matching failure may reflect a vocabulary gap, redundant query words, or spelling errors. It may not be obvious from the number of results that there’s a recall problem, but the missing results negatively impact metrics.
  • Searches where searchers override default ranking, e.g. sorting by price. Searcher-specified sorts generally do not promote relevant results — for example, the lowest-price results are unlikely to be relevant. Hence, search needs to manage a precision-recall tradeoff for the entire result set.

Improving Recall

There are three main ways to improve recall.

  • Query expansion: a reductionist approach that expands words or phrases using a dictionary. This approach is simple, but it does not respect context (e.g., wine glasses -> wine eyeglasses). It works best in conjunction with an approach that ensures precision, such as query categorization. It’s best to keep the dictionary size manageable, as well as to avoid conflating spelling and stemming word variations with semantic synonyms.
  • Query relaxation: instead of expanding words, it relaxes them, i.e., makes them optional. The challenge is choosing which words to relax. Relaxation can drastically affect the precision-recall tradeoff, so it’s important to choose words wisely. Relaxation aims to choose the least important words, but simple statistical measures like inverse document frequency (idf) may not recognize importance in context (e.g., desk and chair -> desk chair).
  • Whole-query expansion: in contrast to dictionary-based query expansion, this holistic strategy expands the query to include the results of similar queries. Whole-query expansion maps the query to an intent and then expands the results based on other queries with similar intent. This approach is powerful: unlike dictionary-based expansion, it doesn’t suffer from loss of context. But it requires having a mapping of queries to embeddings, as well being able to obtain a query’s nearest neighbors.

Summary

Precision is important for the happy path where searchers find great results on the first page using the default ranking. But recall matters too, and not just for queries that would otherwise return few results. Recall is critical for non-default sorts, as well as for computing useful aggregates like facet counts. There are at least three ways to improve recall: dictionary-based query expansion, query relaxation, and whole-query expansion. Search developers invest in all three of these approaches, and not just in better ranking.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store