Interview Questions for Search Relevance Engineers, Data Scientists, and Product Managers

  • What are the trade-offs between using click-through rate (CTR) and conversion rate as search success metrics?
  • How would you expect improving snippet quality to affect your metrics? And how would you go about measuring snippet quality?
  • Describe a situation where a change might increase the mean-reciprocal rank (MRR) of click positions but also increase search abandonment.
  • How do you use searcher behavior to measure a system that offers both autocomplete and a traditional search results page?
  • What are reasons to use explicit human relevance judgments to measure search quality, rather than just measuring behavior? What are the downsides?
  • How do you determine whether a bigram (or, more generally an n-gram) represents a single concept?
  • How do you determine whether two words or phrases are synonyms? In general, how do you find relationships to use for query expansion?
  • How do you implement techniques like stemming and lemmatization, and what trade-offs do you have to manage?
  • How do you handle the tokenization challenges of phone numbers, part numbers, etc., where the spacing searchers use may be different from what’s in the document collection? (cf. this blog post)
  • What are the benefits and drawbacks of replacing a traditional keyword index by one that embeds all of the documents as vectors?
  • Give some examples of query-dependent vs. query-independent search relevance factors, and discuss the implications for computation.
  • Describe and compare two ways (e.g., a simple heuristic and a principled approach) to incorporate query expansion into a relevance model.
  • How can you train a machine-learned ranking model for search? Elaborate on the differences between two approaches, e.g., pointwise and pairwise.
  • Discuss differences between using filtering and ranking to deliver search relevance, as well as the implications for evaluation and training models.
  • How can sorting search results by relevance lead to a lack of diversity in the results? What techniques can you use to ensure result diversity?

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store