Evaluating Search: Measuring Searcher Behavior

Measuring the effectiveness of your search engine is hard. But you have an army of volunteers helping you: your customers. A great way to measure your search engine’s effectiveness is by analyzing searcher behavior.

How searchers interact with your search engine represents a collection of implicit relevance judgments. It can be tricky to derive insight from these implicit judgments, but it’s well worth the effort. In this post, we’ll explore ways to measure searcher behavior.

It’s about ROI — for the Searcher

Searching is a means, not an end in itself. Searchers invest effort in order to complete a task, their return being that they find what they’re looking for. We improve a searcher’s return on investment (ROI) by either increasing their return (the numerator) or decreasing their investment (the denominator).

Measuring Return

We can never be sure that a searcher found what he or she was looking for. But the two strongest indicators of searcher return are clicks and conversions — and conveniently the latter is also a strong measure of return for your business.

Since conversions represent task completion — the searcher not only found something but actually bought it — you might wonder why we even bother measuring clicks. There are at least two good reasons.

First, conversions are sparse. While conversions are a stronger signal of return for the searcher than clicks, learning from them requires a lot of data. More data means more time to collect it, and time is often your scarcest resource.

Second, there may be good reasons for someone to click on a result but not convert. The searcher may have decided to buy the product at a local store, or to take more time to think about the purchase. The outcome may not be good for your business, but the search experience could still have provided a positive return to the searcher.

That said, not all clicks are good for the searcher. The searcher may click on a result, only to find out after clicking that the product isn’t what he or she was looking for. Clicks tend to overestimate return, while conversions underestimate it. That’s why it’s important to look at both.

Measuring Investment

Searchers invest effort to find what they’re looking for. We can’t hope to make the process completely effortless — not unless we learn how to read their minds! But we should strive to minimize the effort they have to exert in order to achieve their desired return.

Click-Through Rate And Conversion Rate

We can think of the act of performing a search as a single unit of investment. It’s a coarse unit, but nonetheless a useful one that allows us to derive two ROI measures: the click-through rate (CTR) as the fraction of searches resulting in a click, and the conversion rate as the fraction of searches resulting in a conversion.

Mean Reciprocal Rank

Not all searches involve the same amount of effort. The searcher has to take time to scan through the results, looking for something relevant.

Hence, a more granular measure of searcher investment is position of the highest-ranked result that the searcher clicks on. In practice, this is inverted to obtain the reciprocal rank, e.g., if the searcher clicks on the 4th result, the reciprocal rank is 0.25. The average of these reciprocal ranks is called the mean reciprocal rank (MRR).

What about the reciprocal rank for searches that doesn’t receive a click? Here there are two options. One is to not consider those searchers for computing MRR. The other is to treat them as clicks at infinity, contributing a reciprocal rank of 0 (i.e., 1 divided by infinity). Both approaches work, but it’s important to know which one you are using — and very important to keep track of CTR if you’re only including searches that receive clicks when you compute MRR.

Characters Typed

Another source of effort is typing. Typing a long search query requires more effort than typing a short search query, especially when the searcher is typing on a phone. On the other hand, autocomplete can significantly reduce the searcher’s typing effort.

But don’t go overboard in your attempt to reduce effort. If the searcher has only typed in one or two characters, it’s unlikely that you’ll be able to come up with the best autocomplete suggestion. It’s better to let the searcher invest more effort than to disappoint the searcher with a suggestion that doesn’t provide a satisfactory return.

No Perfect Measure

We’ve discussed several ways to measure return and investment. Ideally, we’d have a single measure of each, so that we could compute ROI by dividing one by the other. Unfortunately, all of these measures are proxies, and it would be dangerous to rely on any single measure.

For example, we expect conversion to correlate to clicks. But they don’t always correlate. Sometimes searchers learn something after clicking that we might have told them before they clicked, such as whether the product is available or is eligible for free shipping. Sometimes the searcher is doing research and has no intention of buying anything.

As George Box said, “all models are wrong, but some are useful.” It’s a good idea to track a few measures of searcher return and searcher effort. In general, you want to increase the former and decrease the latter. Most of the time, they’ll move in tandem. But pay attention when they don’t, because that’s when you’re likely to obtain the most useful — and counterintuitive — insights.

Previous post: Evaluating Search: Measure It
Next post: Evaluating Search: Using Human Judgement