This paper, titled "Better Retrievers, Worse Labels," uncovers a critical flaw in how we evaluate Retrieval-Augmented Generation (RAG) systems. It reveals that our standard methods for testing whether a model knows when to say "I don't know" are increasingly broken as our retrieval technology improves.
TL;DR
Evaluating RAG abstention using "gold document" shortcuts is fundamentally flawed because better retrievers often find valid answers in unjudged documents. This creates a "false miss" problem that unfairly penalizes high-performing systems, making them appear less capable of detecting unanswerable questions than they actually are.
What Problem Does This Paper Address?
In RAG, a system should only answer if the retrieved documents provide enough information. If they don't, it should abstain (e.g., say "I don't know"). To evaluate this at scale, researchers often use a "shortcut": if the retriever fails to find the specific "gold" documents identified by human annotators, the context is labeled as "insufficient," and the system is expected to refuse to answer.
However, relevance judgments in datasets are notoriously incomplete. A strong retriever might find a perfect answer in a document that the original annotators simply didn't see or label. The authors argue that this shortcut creates a "retriever-dependent bias" that distorts evaluation: the better your retriever is at finding "off-label" relevant info, the worse your system looks according to the standard metrics.
Methods
The researchers conducted a reproducibility study by re-running official baselines for conversational RAG benchmarks. They compared two ways of labeling whether a retrieved context is "sufficient":
- The Shortcut (qrels-derived): A context is labeled insufficient if it contains zero "gold" documents.
- Content-based Labels: The authors used a "panel of judges" (various LLMs like phi-4 and Qwen3-14B) and a human-adjudicated blind audit to determine if the content of the retrieved documents actually supports the answer, regardless of whether they are the official "gold" documents.
They tested these labels across a "ladder" of retrievers, ranging from simple BM25 to advanced hybrid systems, to see how the evaluation changes as retrieval quality improves.
Datasets
The study primarily uses:
- RECOR (RETECO track 2): A reasoning-focused multi-turn conversational retrieval benchmark based on StackExchange and BRIGHT.
- MTRAG: A multi-turn conversational benchmark where relevance judgments come from human annotators interacting with a live RAG agent.
Key Insights and Ideas
- The "Better Retriever" Paradox: As retrievers improve (higher nDCG), the "no gold = insufficient" shortcut becomes increasingly wrong. For the strongest retrievers, up to 72% of the contexts labeled "insufficient" by the shortcut actually contained the correct answer.
- False Misses are Real Answers: When models answer from these "false miss" contexts (where no gold doc is present), they are correct 62–67% of the time. This proves the documents are actually sufficient, and the "gold" labels are simply missing them.
- Evaluation Inversion: Because the shortcut is so flawed, it can make a simple retrieval score (which just measures confidence in finding the specific gold doc) look like a better "abstention detector" than the LLM reader itself. When using content-based labels, the LLM reader's own confidence correctly emerges as the superior detector.
- Where the Answers Hide: In conversational RAG, "missing" answers often come from "hard negatives" (documents about the same topic), documents from other turns in the same dialogue, or different chunks of the same source page.
Key Results
- Precision Collapse: The precision of the "no gold" shortcut as a detector of true insufficiency plummeted from 0.68 for a weak retriever to just 0.10 for a strong hybrid retriever.
- Understated Performance: The standard evaluation metric (AUROC) for abstention was found to be up to 0.19 too low when using the shortcut compared to content-based labels.
- MTRAG Replication: The effect isn't just an artifact of one dataset; on MTRAG, precision dropped from 0.72 to 0.50 as retrieval improved.
Limitations and Future Work
The authors acknowledge that their content-based labels rely heavily on LLM judges. While they validated these with a blind audit and negative controls, there is a risk of "shared leniency" where LLMs might over-credit on-topic but technically insufficient documents.
The authors recommend that future benchmarks:
- Stop using gold-document presence as a proxy for sufficiency when comparing different retrievers.
- Label sufficiency by content for every retrieved context.
- Validate judges with negative controls to ensure they aren't just rewarding "on-topic" noise.
Summary updated