A Statistical Sense of Direction
AI-powered search lives in a world of high-dimensional vectors. But when we care about similarity, we almost always throw away magnitude. We normalize everything and end up working on the surface of a giant unit hypersphere. All that remains is direction.
Modern information retrieval has leaned heavily on statistics for decades, but the moment we start working with unit vectors, the usual toolbox stops fitting. If we want to treat these representations as real statistical objects, we need a branch of statistics that understands directions, angles, and orientation. That branch exists. It is called directional statistics.
Why We Normalize Everything
Most embedding models represent meaning through orientation. Cosine similarity ignores magnitude completely, so the only thing that matters is where a vector points. The hypersphere is not just a geometric metaphor. It is literally the space where our information systems operate.
This is why directional statistics lines up so naturally with modern retrieval. It gives us ways to describe, average, compare, and model data that live on the surface of a hypersphere.
Thinking in Angles, Not Distances
A simple example is the bag-of-documents model. It represents a query as the spherical mean of the relevant document vectors. That mean direction captures what the query is about, translating the query into the document space. The length of the resultant vector, which is related to spherical variance, captures how tightly those documents cluster together, which we interpret as query specificity. A tight cluster indicates a specific query. A loose cluster indicates a broad or ambiguous one.
This is already a substantial upgrade from ad hoc heuristics, and it comes directly from applying directional statistics to a retrieval problem. But this approach barely scratches the surface.
A Mathematical Toolkit
Directional statistics provides a rich set of tools that are directly applicable to embeddings. Here are some examples.
The von Mises–Fisher distribution
The von Mises–Fisher (vMF) distribution is the spherical version of the Gaussian (aka normal) distribution. It has a mean direction and a concentration parameter that controls how tightly points cluster around that direction. A large concentration means the distribution hugs that direction. A concentration of zero means everything is uniformly spread around the unit hypersphere.
It is natural to ask whether a query itself should be modeled as a von Mises–Fisher distribution. Perhaps the concentration parameter captures specificity more naturally than the mean resultant length. A mixture of these distributions might offer a principled way to represent broad and ambiguous queries. It is an appealing direction to explore.
Spherical Medians
If the spherical mean is useful, what about the spherical median? It minimizes the sum of angular distances to it, as opposed to the sum of squared angular distances minimized by the mean. It is more robust to outliers, just as the median is in ordinary statistics. Like a conventional median, it is a bit harder to compute and sometimes not unique, but robustness often matters more than convenience. It may give us a better query representation when relevance sets are noisy or skewed.
Higher-Order Moments
We already use the spherical equivalents of mean and variance, which are the first two moments. What about the spherical analogs of the third and fourth moments: skewness and kurtosis? Skewness tells us whether and how much a distribution leans asymmetrically in one direction. Kurtosis tells us whether the distribution has heavy or light tails. When our vectors represent meaning, asymmetry or tail behavior may capture semantic structure that mean and variance miss.
None of this is exotic. They are basic statistical concepts adapted to the spherical geometry we already work in.
Where This Perspective Can Take Us Next
Embedding-based systems encode meaning as direction, but many of our intuitions are tuned for Euclidean rather than spherical spaces.
Evaluation metrics also suffer. Traditional precision and recall treat relevance as a binary event and ignore similarity structure. When systems operate in a directional space, that simplification hides important distinctions. A directional understanding makes it easier to reason about similarity in terms of substitutability and to measure similarity-sensitive diversity when evaluating search results. These are all areas that benefit from a directional statistics framework.
Search is no longer about counting. It is about aligning. Cosine similarity is the fundamental operation. Directional statistics is the framework that explains it and extends it.
We have already moved our data onto the hypersphere. It is time to move our thinking there too and take search in the right statistical direction.
