Putting Search Ranking in Perspective

For feeds and recommendations, ranking is critical. All the inputs are implicit, so machine-learned ranking is the only practical way to optimize engagement.

Search is different.

Foremost, ranking should respect query understanding.

Ranking should focus primarily on query-independent signals.

To a first approximation, query understanding and the relevance model establish relevance as a binary filter. That allows ranking to focus primarily on query-independent signals of desirability, like popularity, and secondarily on user-dependent signals that can be used for personalization.

Two reasons to use query-dependent signals for ranking:

  • Prototypicality. For example, a query can be associated with a category or price distribution of results. There can be query-dependent prototypicality signals reflecting how a result fits into these distributions.
  • Non-binary relevance. There may be gradations of relevance — particularly when a search engine uses query relaxation to increase recall. The relevance model score can serve as a query-dependent ranking signal.

But don’t invest in query-dependent ranking signals until you have at least established a robust relevance model and strong query-independent signals.

A ranking model only learn from signals that searchers see or infer.

But a model cannot learn from signals that searchers can neither see nor infer. If a signal is not available on the search results page, it cannot influence the searcher’s decision to click on a result. If it is not available on the listing page, then it cannot even influence further actions, like purchases.

You can use these invisible signals for ranking. But they cannot be learned from searcher behavior. Instead, you determine their contribution through hand-crafted business rules, e.g., demoting products with high return rates.

A/B-testing is the gold standard, but offline evaluation is possible.

But you can still perform an offline evaluation as a sanity check. You can replay search logs, re-rank the displayed results (i.e., the first page) using the new model scores and see how the new positions of reranked results affect metrics like the mean reciprocal rank (MRR) of clicks.

If the replay analysis shows an improvement, then the new ranking model is positive — at least when it is used as a re-ranker. If not, it’s still possible that it’s an improvement — but that you cannot learn from the replay because of presentation bias. After all, searchers can only engage with the results they see. Nonetheless, a neutral or negative result from reply should at least make us skeptical as you decide how to prioritize an A/B test.

Summary: keep ranking in perspective, and rank wisely.



High-Class Consultant.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store