Better Retrievers, Worse Labels: How Incomplete Relevance Judgments Distort Abstention Evaluation in Conversational RAG

Ivan Kochergin, Vasily KonovalovarXiv 2026 · arXiv

This paper, titled "Better Retrievers, Worse Labels," uncovers a critical flaw in how we evaluate Retrieval-Augmented Generation (RAG) systems. It reveals that our standard methods for testing whether a model knows when to say "I don't know" are increasingly broken as our retrieval technology improves.

TL;DR

Evaluating RAG abstention using "gold document" shortcuts is fundamentally flawed because better retrievers often find valid answers in unjudged documents. This creates a "false miss" problem that unfairly penalizes high-performing systems, making them appear less capable of detecting unanswerable questions than they actually are.

What Problem Does This Paper Address?

In RAG, a system should only answer if the retrieved documents provide enough information. If they don't, it should abstain (e.g., say "I don't know"). To evaluate this at scale, researchers often use a "shortcut": if the retriever fails to find the specific "gold" documents identified by human annotators, the context is labeled as "insufficient," and the system is expected to refuse to answer.

However, relevance judgments in datasets are notoriously incomplete. A strong retriever might find a perfect answer in a document that the original annotators simply didn't see or label. The authors argue that this shortcut creates a "retriever-dependent bias" that distorts evaluation: the better your retriever is at finding "off-label" relevant info, the worse your system looks according to the standard metrics.

Methods

The researchers conducted a reproducibility study by re-running official baselines for conversational RAG benchmarks. They compared two ways of labeling whether a retrieved context is "sufficient":

  1. The Shortcut (qrels-derived): A context is labeled insufficient if it contains zero "gold" documents.
  2. Content-based Labels: The authors used a "panel of judges" (various LLMs like phi-4 and Qwen3-14B) and a human-adjudicated blind audit to determine if the content of the retrieved documents actually supports the answer, regardless of whether they are the official "gold" documents.

They tested these labels across a "ladder" of retrievers, ranging from simple BM25 to advanced hybrid systems, to see how the evaluation changes as retrieval quality improves.

Datasets

The study primarily uses:

  • RECOR (RETECO track 2): A reasoning-focused multi-turn conversational retrieval benchmark based on StackExchange and BRIGHT.
  • MTRAG: A multi-turn conversational benchmark where relevance judgments come from human annotators interacting with a live RAG agent.

Key Insights and Ideas

  • The "Better Retriever" Paradox: As retrievers improve (higher nDCG), the "no gold = insufficient" shortcut becomes increasingly wrong. For the strongest retrievers, up to 72% of the contexts labeled "insufficient" by the shortcut actually contained the correct answer.
  • False Misses are Real Answers: When models answer from these "false miss" contexts (where no gold doc is present), they are correct 62–67% of the time. This proves the documents are actually sufficient, and the "gold" labels are simply missing them.
  • Evaluation Inversion: Because the shortcut is so flawed, it can make a simple retrieval score (which just measures confidence in finding the specific gold doc) look like a better "abstention detector" than the LLM reader itself. When using content-based labels, the LLM reader's own confidence correctly emerges as the superior detector.
  • Where the Answers Hide: In conversational RAG, "missing" answers often come from "hard negatives" (documents about the same topic), documents from other turns in the same dialogue, or different chunks of the same source page.

Key Results

  • Precision Collapse: The precision of the "no gold" shortcut as a detector of true insufficiency plummeted from 0.68 for a weak retriever to just 0.10 for a strong hybrid retriever.
  • Understated Performance: The standard evaluation metric (AUROC) for abstention was found to be up to 0.19 too low when using the shortcut compared to content-based labels.
  • MTRAG Replication: The effect isn't just an artifact of one dataset; on MTRAG, precision dropped from 0.72 to 0.50 as retrieval improved.
Figure 2: (a) Share of top-5 contexts labelled insufficient by the qrels label (no gold) and by the two content judges (no; partial counted as sufficient), against first-stage nDCG@10. (b) Precision of “no gold” as a detector of content insufficiency under each labeller; the audit covers 50 NONE contexts per retriever; grey: the four small judges of

Limitations and Future Work

The authors acknowledge that their content-based labels rely heavily on LLM judges. While they validated these with a blind audit and negative controls, there is a risk of "shared leniency" where LLMs might over-credit on-topic but technically insufficient documents.

The authors recommend that future benchmarks:

  1. Stop using gold-document presence as a proxy for sufficiency when comparing different retrievers.
  2. Label sufficiency by content for every retrieved context.
  3. Validate judges with negative controls to ensure they aren't just rewarding "on-topic" noise.

Summary updated

Abstract

Conversational RAG benchmarks such as MTRAG and RETECO evaluate retrieval and generation end to end: each system answers from its own retrieval, so abstention must be scored against whether each retrieved context is sufficient. A convenient shortcut reads sufficiency off the relevance judgments and treats a context without gold documents as unanswerable. We re-run the official baselines exactly and test this shortcut on RECOR (RETECO track~2) with four retrievers (nDCG@10 from .18 to .59) and three Qwen3 readers. Two open LLM judges and a panel of four small open models label each context by content, and a model of another family audits 250 contexts blind, with adjudication. The stronger the retriever, the larger the share of no-gold contexts that still contain the answer (28\% to 72\%) and the lower the shortcut's precision (.68 to .10; .70 to .21 under the audit). A reader agrees: Qwen3-4B answers 62--67\% of these false misses correctly, against 23--30\% of no-gold contexts judged insufficient and 1\% with no documents. In most false misses, the answer comes from a hard negative mined for the same question, another turn's gold or a neighbouring chunk of the gold page. On MTRAG, whose judgments come from annotators, precision drops from .72 to .50. Under the shortcut, answerability AUROC is up to .19 too low, and the best sufficiency detector appears to be a retrieval score that never reads the documents. We release code, runs and labels, and recommend scoring refusals against content labels, not the presence of gold.
BibTeX
@misc{kochergin2026better,
  author = {Ivan Kochergin and Vasily Konovalov},
  title = {{Better Retrievers, Worse Labels: How Incomplete Relevance Judgments Distort Abstention Evaluation in Conversational RAG}},
  year = {2026},
  eprint = {2610.40966},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40966},
}
Figure