Measuring Document Relevance Through Model Uncertainty

20 concepts13 papers2 proposals

Traditional search engines and retrieval-augmented generation (RAG) pipelines share a common assumption: document relevance can be compressed into a single, deterministic scalar score. Whether it is a BM25 lexical match score, a dense vector cosine similarity, or a neural re-ranker logit, the system treats this number as an absolute truth.

However, relevance is inherently uncertain. A high similarity score might be a fluke of data noise, an artifact of an ambiguous query, or a misleading "distractor" document that looks topically related but fails to answer a user's question. By forcing a model to output a single point estimate, we throw away crucial information about how confident the model actually is in its prediction.

A growing body of research flips this paradigm. Instead of treating uncertainty as a problem to be eliminated, these approaches leverage a model's internal uncertainty—or represent relevance explicitly as a probability distribution—to build more robust, noise-resistant retrieval systems.

Relevance as a Probability Distribution

The shift away from scalar scores began with the realization that noisy data sources require a distributional representation. In early search evaluation, Carterette et al. demonstrated that modeling relevance as a multinomial distribution rather than a fixed scalar allows systems to calculate statistical confidence intervals over noisy user click logs. Similarly, Wang et al. showed that treating click-based relevance as a probability distribution captures the nuanced preferences of users far better than deterministic values, leading to more robust ranking models.

With the advent of deep learning, this distributional perspective moved directly into neural architectures. Instead of relying on a single forward pass to output a score, Cohen et al. applied Monte Carlo (MC) Dropout to deep retrieval models. By sampling the network multiple times, they generated an entire distribution of scores for each document. They discovered a powerful correlation: highly relevant documents produce low-variance, highly skewed score distributions, whereas irrelevant documents produce chaotic, high-variance distributions.

Deterministic Scoring:
Query + Document ───► [ Neural Model ] ───► Score: 0.85 (Is it a confident 0.85 or a wild guess?)

Distributional Scoring (MC Dropout / Parametric):
Query + Document ───► [ Neural Model ] ───► Mean: 0.85, Variance: 0.02 (Confident Match)
Query + Distractor ──► [ Neural Model ] ───► Mean: 0.81, Variance: 0.45 (Uncertain/Noisy Match)

Modern listwise and pairwise re-rankers formalize this by predicting parametric distributions directly. Frameworks proposed by Yu et al., Wang et al., and Yoon et al. model document scores as Gaussian distributions with learnable means and variances. This allows a system to distinguish between a document with a stable, high-confidence rank and one that requires further evaluation by a more expensive model, optimizing both ranking accuracy and computational efficiency.

The Intuition of Generative Uncertainty

Why does a language model's internal uncertainty serve as such a precise proxy for document relevance? To understand the intuition, consider how a generative model processes information.

Imagine asking a Large Language Model (LLM) a specific question, like "What year was the treaty signed?" * If you provide the model with an irrelevant distractor document, the model's internal probability space remains flat. It has to guess among many possible dates, resulting in high token-level entropy. It might even exhibit a high probability of generating an "unknown" or "unanswerable" token. * If you provide a highly relevant document that explicitly states the answer, the document acts as a constraint that collapses the probability space. The model's confidence concentrates heavily on the correct tokens, causing its internal predictive entropy to plummet.

This behavior allows us to use the model's own internal state as an unsupervised relevance filter. As Lesota et al. demonstrated in deep generative retrieval, term-level entropy during query generation serves as a direct indicator of relevance: lower confidence in generating a query from a document correlates strongly with low document relevance.

In open-domain question answering, Xin et al. leveraged this by tracking the model's internal probability of generating an "unknown" response. By using this unsupervised uncertainty signal to re-rank retrieved content, systems can successfully filter out distractor documents before they corrupt the final answer generation.

Measuring Relevance via Information Gain

Instead of relying on external vector similarity metrics—which only measure surface-level semantic overlap—modern retrieval-augmented generation (RAG) frameworks evaluate a document's relevance based on its Information Gain. True utility is measured by how much a document changes the model's confidence from a state of ignorance to a state of certainty.

Query: "Where was the treaty signed?"

Context A (Distractor): "The treaty was signed after long negotiations between both parties..."
LLM Output Distribution: [Versailles: 20%, Paris: 15%, London: 10%, Unknown: 35%] 
Result: High Entropy / High Uncertainty ──► Document A is Irrelevant

Context B (Gold Standard): "The treaty was signed in the Hall of Mirrors at Versailles..."
LLM Output Distribution: [Versailles: 92%, Paris: 2%, London: 1%, Unknown: 1%]
Result: Low Entropy / Low Uncertainty ──► Document B is Highly Relevant

This mechanism forms the core of several recent breakthroughs:

  • Logit-Based Uncertainty Reduction: The SGIC framework, introduced by Chen et al., estimates document relevance by measuring the literal reduction in an LLM's predictive uncertainty (derived from internal logit distributions) when a specific document is added to the context window. Similarly, the CLeHe strategy proposed by Qiu et al. uses output entropy to weight information across multiple documents, contrasting low-entropy states against high-entropy layers to bypass noise.
  • Empirical Information Gain: The InfoGain-RAG framework by Wang et al. explicitly quantifies this contribution as "Document Information Gain," training specialized re-rankers to prioritize documents that maximize the LLM's confidence in generating a correct answer.
  • Semantic Consistency: When logit distributions are inaccessible (such as with closed-source APIs), uncertainty can be measured via generation consistency. The SUNAR framework, developed by Venktesh et al., generates multiple candidate answers from a single document. If the generated answers vary wildly, the document exhibits high semantic uncertainty and is flagged as a distractor. If the answers are highly consistent, the document is deemed reliable. This aligns with the work of Cho et al., which pairs query-relevance scoring with unanswerability constraints to ground the reader and penalize overconfident, topically unrelated answers.

By shifting from static scalar matching to dynamic, uncertainty-aware estimation, these techniques ensure that retrieval systems are tightly coupled with the downstream model's actual reasoning needs. A document is no longer judged merely by whether it looks like the query, but by its proven ability to resolve the model's doubt.

Key Foundational Work

Explore click models for search ranking (2010) — contributed 3 concept elements to this direction.
Evaluating Search Engines by Modeling the Relationship Between Relevance and Clicks (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_uncertainty_filtering_d17060eb,
  title = {semantic uncertainty filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-uncertainty-filtering/rdr_d17060ebdd934a9ba8d2a4b8752fe275},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_uncertainty_filtering_d17060eb_v1,
  title = {semantic uncertainty filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-uncertainty-filtering/rdr_d17060ebdd934a9ba8d2a4b8752fe275/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake