Evaluating Search: Measuring Searcher Behavior

Measuring the effectiveness of your search engine is hard. But you have an army of volunteers helping you: your customers. A great way to measure your search engine’s effectiveness is by analyzing searcher behavior.

How searchers interact with your search engine represents a collection of implicit relevance judgments. It can be tricky to derive insight from these implicit judgments, but it’s well worth the effort. In this post, we’ll explore ways to measure searcher behavior.

It’s about ROI — for the Searcher

Measuring Return

Since conversions represent task completion — the searcher not only found something but actually bought it — you might wonder why we even bother measuring clicks. There are at least two good reasons.

First, conversions are sparse. While conversions are a stronger signal of return for the searcher than clicks, learning from them requires a lot of data. More data means more time to collect it, and time is often your scarcest resource.

Second, there may be good reasons for someone to click on a result but not convert. The searcher may have decided to buy the product at a local store, or to take more time to think about the purchase. The outcome may not be good for your business, but the search experience could still have provided a positive return to the searcher.

That said, not all clicks are good for the searcher. The searcher may click on a result, only to find out after clicking that the product isn’t what he or she was looking for. Clicks tend to overestimate return, while conversions underestimate it. That’s why it’s important to look at both.

Measuring Investment

Click-Through Rate And Conversion Rate

Mean Reciprocal Rank

Hence, a more granular measure of searcher investment is position of the highest-ranked result that the searcher clicks on. In practice, this is inverted to obtain the reciprocal rank, e.g., if the searcher clicks on the 4th result, the reciprocal rank is 0.25. The average of these reciprocal ranks is called the mean reciprocal rank (MRR).

What about the reciprocal rank for searches that doesn’t receive a click? Here there are two options. One is to not consider those searchers for computing MRR. The other is to treat them as clicks at infinity, contributing a reciprocal rank of 0 (i.e., 1 divided by infinity). Both approaches work, but it’s important to know which one you are using — and very important to keep track of CTR if you’re only including searches that receive clicks when you compute MRR.

Characters Typed

But don’t go overboard in your attempt to reduce effort. If the searcher has only typed in one or two characters, it’s unlikely that you’ll be able to come up with the best autocomplete suggestion. It’s better to let the searcher invest more effort than to disappoint the searcher with a suggestion that doesn’t provide a satisfactory return.

No Perfect Measure

For example, we expect conversion to correlate to clicks. But they don’t always correlate. Sometimes searchers learn something after clicking that we might have told them before they clicked, such as whether the product is available or is eligible for free shipping. Sometimes the searcher is doing research and has no intention of buying anything.

As George Box said, “all models are wrong, but some are useful.” It’s a good idea to track a few measures of searcher return and searcher effort. In general, you want to increase the former and decrease the latter. Most of the time, they’ll move in tandem. But pay attention when they don’t, because that’s when you’re likely to obtain the most useful — and counterintuitive — insights.

Previous post: Evaluating Search: Measure It
Next post:
Evaluating Search: Using Human Judgement



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store