Ranking vs. Relevance

Most folks who work on search worry about relevance. But it’s surprisingly difficult to find a useful definition of relevance.

Defining Relevance

William Goffman defines it as “a measure of information conveyed by a document relative to a query…[but] the relationship between the document and the query, though necessary, is not sufficient to determine relevance.”

These strike me less as definitions and more as an “I know it when I see it” standard. But they’ll have to do.

Ranking

This ranking approach tends to quash diversity: it assigns similar scores to similar results, leading to homogeneity in the top-ranked results. A common technique to increase diversity is reranking top results: demoting near-duplicates or optimizing for some target distribution.

Regardless of how it is implemented, ranking optimizes for a function that reflects searcher and business objectives. For example, ecommerce sites optimize for clicks and purchases — objectives that mostly align the interests of shoppers and retailers. Sometimes there are conflicts of interest, e.g., when the shopper hopes to spend less and the retailers hopes the shopper will spend more. But effective ranking is mostly a win-win.

Ranking vs. Relevance

As noted earlier, retrieval effectively serves as the most significant bit of the ranking score. If the search engine only retrieves relevant results, then relevance — modeled here as binary — acts as this most significant bit. Relevance isn’t binary, but modeling it as binary is a good approximation — especially for applications where searchers override the default ranking, such as sorting by price.

But, if relevance provides the most significant bit, what about the rest of the score? Once the search engine has established relevance, ranking should mostly focus on query-independent signals, such as quality or popularity. To a lesser degree, ranking can take into account prototypicality and non-binary relevance — recognizing that even if all the retrieved results are relevant, some may be more relevant than others.

Ranking is also an opportunity to make tradeoffs between searcher and business objectives. Promoted search results play an important role in the search ecosystem, and the ranker is responsible for using them appropriately.

Summary

Relevance and ranking are crucial concerns— together with recall, they represent the 3 Rs of search. But remember to keep ranking in perspective.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store