New Choice

In improv, there’s a fun game called “new choice”. At any point during the improvised scene, the host can call for a “new choice”. The player who last spoke has to backtrack and substitute a new line. At its best, the substitution takes the scene in a new, unexpected, and hilarious direction.

Interacting with a search engine or a digital assistant probably shouldn’t feel like watching — or participating in — improvisational comedy. But I often wish I could play “new choice” to nudge the information-seeking experience in a different direction.

Relevance Feedback

But there’s really no substitute for explicit feedback. Inferring relevance judgements from behavior is at best delayed — by the time you’ve abandoned the search session, it’s a bit late to ask “wait, wait, why didn’t you click on that?” — and at worst misleading, since the reasons for engagement or lack thereof may have nothing to do with relevance. Pseudo-relevance feedback, despite being surprisingly effective, does not take any input from the searcher, but rather doubles down on the relevance model — as well as its biases. Only explicit feedback takes real — and real-time — input from the searcher and creates the opportunity for the search engine to act directly on that input.

Usability Challenges

  • Asymmetric feedback. When searchers find what they are looking for, they’re unlikely to make an extra effort to provide explicit feedback. On the other hand, when searchers can’t find what they want, they may be more inclined to provide feedback, especially if it leads to different results.
  • Underspecified feedback. When searchers provide explicit feedback about a result, it’s likely to be binary, e.g., thumbs up or down, or perhaps swiping left or right. Unfortunately, binary feedback doesn’t indicate what a searcher liked or didn’t like about the result. Since obtaining non-binary feedback (e.g., asking why) adds additional complexity to the interface, it may be better to collect multiple data points to infer a pattern.
  • Managing expectations. When searchers provide explicit feedback, they expect the search engine to listen to them. Specifically, they expect the search engine to immediately act on their feedback, in the context of their current search session. Many search engines are unable to incorporate this real-time feedback; other search engines may overreact to it.

These are some of the key usability challenges we have to address in order to make use of explicit relevance feedback.

Baby Steps

Here are some ideas that directly address the above usability challenges:

  • Embrace diversity. Searchers are more able and willing to provide binary feedback (e.g., “more like this”, “less like this”) when they see a diversity of results. Indeed, explicit relevance feedback works best for broad and ambiguous queries, so make sure to convey that breadth or ambiguity. That way, searchers have a sense of the intent space they are navigating.
  • Get lots of binary feedback. It’s tempting to ask searchers for rich feedback, but it’s better to ask simple questions — lots of them. Just as an eye exam arrives at a precise prescription through a series of binary questions, we can use binary questions to solicit preferences. The tricky part is including enough redundancy to establish confidence in a pattern, while including enough variety to cover the preference space — all while not wearing out searchers’ patience. But let’s at least keep it simple!
  • Do something. When searchers provide feedback, they expect something to happen — preferably immediately. Negative feedback on a result should, at the very least, remove that result from view. After sufficient feedback, searchers should see changes to the results that reflect their input. The interface needs to set expectations about how much feedback is sufficient, e.g., 5 to 10 judgments, and then meet those expectations with action.

Search is a communication process.



High-Class Consultant.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store