Search engines are at the heart of the digital world. We interact with them on the outside, using search engines to find information, products, and more. But they do even more unseen work behind the firewall, processing event logs, serving as schemaless databases, etc. So it often surprises me how…

There’s a lot of writing about search — especially about ranking and relevance. And recently there’s been an increased focus on the particular challenges of query understanding. But, surprisingly, there hasn’t been much discussion of content understanding. This post will be the first of many to address this gap. Content understanding is the foundation of the search process. Let’s…

Many platforms, particularly social media sites and ecommerce marketplaces, establish policies to determine what they consider acceptable content. These platforms implement such policies through content moderation systems. Values and Tradeoffs The policy underlying a content moderation system reflects a particular set of values. When there are conflicts among those values, the policy provides…

Despite its importance, content quality is hard to define without devolving into circularity. What we mean by content quality is a measure or collection of measures that represents the content’s value independent of any searcher’s particular information need. In other words, content quality complements relevance. While relevance measures how well…

One of the challenges facing search applications is that users aren’t necessarily searching for documents. To put it another way, the unit of content that a user is looking for doesn’t necessarily correspond to a unit of content in the search index. Given this potential mismatch, it’s useful to understand…

Earlier this year, Grant Ingersoll and I kicked off a 4-week co:rise course on Search with Machine Learning. We covered search fundamentals, and then went on to teach ranking (both hand-tuned and LTR), content understanding, and query understanding. Students worked with a real (if slightly dated) ecommerce product catalog, developing…

Content classification and annotation offer useful approaches for content understanding, recognizing whether a piece of content is about a particular topic or mentions a particular entity. But most content exists in a space that is too rich to reduce to classification and annotation. A document is more than a category…

Content understanding requires both holistic and reductionist approaches, just like query understanding. The previous post focused on content classification as a holistic approach. This post explores content annotation as a reductionist approach. While content classification assigns labels to the content as a whole, content annotation focuses on specific words or…