When we use a search engine, we look for documents that are topically relevant to our query. For decades, the field of Information Retrieval (IR) has optimized for this exact human preference, using keyword overlap (like BM25) or dense vector spaces to maximize semantic similarity.
However, when a Large Language Model (LLM) is the consumer of retrieved information—such as in Retrieval-Augmented Generation (RAG), in-context learning, or tool selection—this definition of relevance breaks down. An LLM does not read like a human; it processes text to minimize next-token prediction loss and resolve conditional probabilities. This divergence creates a fundamental mismatch: a document can be perfectly "on-topic" according to a retriever, yet possess zero utility for the downstream generator.
As noted by Hengran Zhang et al. in Are Large Language Models Good at Utility Judgments?, traditional retrieval systems excel at identifying passages that share a topic with a query, but frequently fail to provide the specific evidence required to resolve an information need. This discrepancy is known as the relevance-utility gap.
Why Semantic Similarity Fails the Generator
To understand why the relevance-utility gap occurs, we must look at the mathematical disconnect between retrieval and generation objectives. A dense retriever typically optimizes the cosine similarity between a query embedding $e_q$ and a document embedding $e_d$. In contrast, a generator maximizes the conditional probability of a target response $Y$ given the prompt $X$ and retrieved context $D$: $P(Y|X, D)$.
This disconnect manifests in several ways:
- Relevant Distractors: A document that shares high semantic vector similarity with a query might simply repeat the query's premises or introduce redundant, superficial context. Yutao Zhu et al. demonstrate in Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation that highly similar documents can act as noise, containing misleading information that causes reasoning failures, while documents with lower similarity scores may contain the exact inferential link the model needs.
- Structural and Architectural Biases: LLMs are sensitive to how information is structured and where it is placed. Youngwon Lee et al. in Inference Scaling for Bridging Retrieval and Augmented Generation highlight that highly relevant documents can actually degrade answer quality depending on their position within a prompt, meaning a retriever's ranking does not inherently correspond to generation success.
- Domain-Specific Preference Gaps: In specialized tasks like code generation, lexical or semantic similarity is an unreliable proxy for helpfulness. Xinyu Gao et al. observe in Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation that the most semantically similar code snippets often lack the precise structural patterns or novel knowledge required by the downstream generator to complete a programming task.
Quantifying Utility: Information Gain and Adequacy
To bridge this gap, recent research shifts the focus from static similarity metrics toward dynamic, model-centric measures of utility. Instead of asking "how close are these two vectors?", researchers ask "how much does this document help the model generate the correct answer?"
One prominent approach is to directly quantify a document's contribution to the model's confidence. Zihan Wang et al. introduce this perspective in InfoGain-RAG: Boosting Retrieval-Augmented Generation through Document Information Gain-based Reranking and Filtering by proposing a metric called Document Information Gain (DIG). DIG measures the actual shift in the LLM's generation confidence when a document is added to the context, allowing a specialized reranker to filter out shallow semantic matches that offer no new information.
Traditional RAG: [Query] ──(Semantic Similarity)──> [Top-K Documents] ──> [LLM]
│
┌─────────────────────────────────────┘
▼
Utility-Aligned: [Query] ──> [Reranker trained on Document Information Gain] ──> [LLM]
Similarly, the concept of "document adequacy" replaces topical alignment with evidential completeness. As argued by Dongyang Zeng et al. in EAReranker: Efficient Embedding Adequacy Assessment for Retrieval Augmented Generation, evaluating adequacy ensures that the retrieved text provides the necessary logical framework and verifiable structure to support the downstream generation task, rather than just matching keywords.
Closing the Loop via Feedback-Driven Retrieval
If the generator is the ultimate judge of utility, the most effective way to optimize a retriever is to let the generator guide it. This is achieved by treating the language model's internal states, generation loss, or outputs as a direct feedback loop.
In RetGen: A Joint Framework for Retrieval and Grounded Text Generation Modeling, Yizhe Zhang et al. propose an end-to-end framework that jointly trains a dense retriever and a text generator. By using the language model's own predictive signal as a reward, the retriever learns to bypass standard semantic similarity constraints and instead pull documents that provide the precise linguistic or factual cues required for generation.
Taking this a step further, Siyuan Chen et al. develop GRO-RAG: Gradient-aware Re-rank Optimization for Multi-source Retrieval-Augmented Generation, a training-free framework that leverages the language model's gradients. By calculating how effectively individual documents minimize generation loss during a single backward pass, the system dynamically identifies and prioritizes the most useful context from multiple sources.
Alternatively, models can monitor their own internal states to determine when retrieval is necessary. Zijun Yao et al. demonstrate this in SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented Generation, where the LLM evaluates its own hidden-layer uncertainty. Instead of relying on external heuristic relevance scores, the model judges whether a retrieved document genuinely resolves its specific internal knowledge gaps.
By moving from relevance to utility, this paradigm shift ensures that retrieval systems are no longer optimized in isolation, but are explicitly aligned with the internal reasoning mechanisms of the models they serve.