Retrieval and Similarity in the Probability Simplex

23 concepts16 papers1 proposal

When we compare two documents, queries, or text snippets, our default instinct is often to treat them as vectors in a flat, unconstrained geometric space. We compute the angle between them using cosine similarity or find the straight-line distance via Euclidean distance.

However, when text is modeled probabilistically—such as word distributions in language models or topic proportions in latent topic modeling—this flat geometric assumption breaks down. Probability vectors do not live in an infinite, unconstrained space; they live on a bounded probability simplex where all dimensions must be non-negative and sum to exactly one.

To accurately capture text similarity in this space, a prominent research direction focuses on distributional divergence retrieval: treating text as a probability distribution and measuring similarity using information-theoretic divergence metrics rather than geometric distances.

The Flaw of Flat Geometry

Why do standard geometric metrics fail on probabilistic text representations? Consider a topic model that outputs a distribution over themes for a given document. If we use cosine similarity or Euclidean distance to compare these vectors, we treat each topic as an orthogonal, independent coordinate axis.

As Yukari Ogura et al. observe in Text Classification based on the Latent Topics of Important Sentences extracted by the PageRank Algorithm, geometric measures are fundamentally suboptimal for probability densities. Geometric distance treats a change in probability from $0.01$ to $0.11$ identically to a change from $0.40$ to $0.50$. Yet, in a probability space, the first case represents a tenfold increase in relative likelihood, while the second is a minor shift.

Furthermore, flat metrics ignore the curved geometry inherent to probability spaces. Hal Daumé III demonstrates this limitation in Markov Random Topic Fields, showing that when regularizing or penalizing differences between topic distributions across connected documents in a graph, standard Euclidean distance fails to capture true topical similarity. It lacks the sensitivity required to handle the boundaries of the simplex, where probabilities approach zero.

Text as a Statistical Distribution

To leverage information theory, we must first frame text retrieval and matching tasks as comparisons between statistical distributions. This framing manifests across several distinct text-processing domains:

  • Information Retrieval (IR): Instead of matching keywords, the retrieval process can be modeled as measuring the distance between a query distribution and a document distribution. As Eric Gaussier explains in his comprehensive review of Statistical Language Models for Information Retrieval, this framework allows systems to handle complex query models—such as those expanded via user relevance feedback—by directly comparing two distinct probability distributions rather than assuming one simply generated the other.
  • Cross-Document Coreference: Identifying whether two different mentions refer to the same entity requires deep contextual understanding. Chung H. Gooi et al. (Cross-Document Coreference on a Large Scale Corpus) and Si Li et al. (PRIS at TAC 2009: Experiments in KBP Track) show that treating the surrounding context of an entity mention as a probability distribution of terms provides a far more robust signal for identity than basic term overlap, capturing subtle domain and genre patterns.
  • Topic Dynamics and Segmentation: When tracking how a conversation or news broadcast shifts over time, text blocks can be mapped to latent topic distributions. Mi-Mi Lu et al. utilize this in Probabilistic Latent Semantic Analysis for Broadcast News Story Segmentation to detect boundaries where one news story ends and another begins based on sudden shifts in the underlying topic distribution.

The Divergence Toolbox

Once text is framed as a distribution, researchers swap out geometric metrics for information-theoretic ones. The choice of divergence metric depends heavily on the specific constraints of the task.

Kullback-Leibler (KL) Divergence and Cross-Entropy

KL divergence measures the "relative entropy" or inefficiency of using one distribution to approximate another. Zhi-Hua Zhou et al. leverage this in Exploiting Image Contents in Web Search to capture the nuances of semantically expanded text representations. Similarly, Jianguang Du et al. demonstrate in Topic Modeling with Document Relative Similarities that symmetric KL divergence significantly outperforms squared Euclidean distance when learning document relationships from human relative constraints.

Jensen-Shannon (JS) Divergence

A primary drawback of KL divergence is that it is asymmetric and goes to infinity if a word or topic has a probability of zero in one of the distributions. To circumvent this, researchers turn to Jensen-Shannon divergence, which computes a symmetric, smoothed distance between two distributions and their average.

David W. Patterson et al. demonstrate in Sophia: A novel approach for Textual Case-based Reasoning that JS divergence offers a highly robust alternative in high-dimensional text spaces. This sentiment is echoed by Lucas Sterckx et al. (Topical Word Importance for Fast Keyphrase Extraction), who note that relying on cosine similarity can overlook nuanced distributional relationships that JS divergence naturally uncovers. In fraud detection, Vlad Sandulescu et al. (Detecting Singleton Review Spammers Using Semantic Similarity) use symmetric distributional overlap metrics to establish precise thresholds for identifying duplicate, rephrased reviews written by the same spammer.

Hellinger and Bhattacharyya Distances

Unlike KL divergence, Hellinger and Bhattacharyya distances satisfy the triangle inequality, making them true mathematical metrics that respect the curved geometry of the probability simplex. Ugochukwu Onyepunuka et al. use Hellinger distance in Multilingual Analysis of YouTube's Recommendation System... to precisely quantify how far recommended video topics drift from an initial seed video across successive recommendation depths.

Overcoming the Computational Bottleneck

If distributional distances are so much more accurate for text, why aren't they used everywhere? The bottleneck is computational complexity. Standard vector spaces allow for sublinear approximate nearest-neighbor (ANN) search using algorithms like Locality-Sensitive Hashing (LSH) or k-d trees. Most divergence metrics do not natively support these fast search structures.

To bridge this gap, researchers have developed mathematical transformations that map probability spaces back into efficient geometric spaces without losing their distributional benefits.

[Probability Simplex] ──(Hellinger Transformation)──> [Unit Sphere / Euclidean Space] ──> [Fast ANN Search (LSH / k-d trees)]

For example, Kriste Krstovski et al. tackle this directly in Efficient Nearest-Neighbor Search in the Probability Simplex. They demonstrate that by calculating the square root of the probabilities, the Hellinger divergence can be mathematically reduced to standard Euclidean distance on a unit sphere. This elegant mapping unlocks the use of fast, sublinear ANN search techniques, providing massive retrieval speedups with virtually no loss in accuracy.

A parallel approach adapts continuous distributions to dense vector retrieval. In Distribution-Driven Dense Retrieval: Modeling Many-to-One Query-Document Relationship, Junfeng Kang et al. model documents as Gaussian distributions rather than single static vectors to capture multi-aspect meanings. They prove that when the covariance matrix of the document distribution is constrained to an identity matrix, the probabilistic similarity metric naturally simplifies to negative Euclidean distance.

By designing frameworks where standard geometric retrieval emerges as a special case of a broader probabilistic distance, the field successfully marries the semantic richness of the probability simplex with the blazing speed of modern vector search engines.

Key Foundational Work

Sophia: A novel approach for Textual Case-based Reasoning (2005) — contributed 1 concept elements to this direction.
Cross-Document Coreference on a Large Scale Corpus (2004) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_distributional_distance_retrieval_9ca791f0,
  title = {distributional distance retrieval},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distributional-distance-retrieval/rdr_9ca791f088514002b149566739af7d13},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_distributional_distance_retrieval_9ca791f0_v1,
  title = {distributional distance retrieval},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distributional-distance-retrieval/rdr_9ca791f088514002b149566739af7d13/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake