When a system answers questions by first retrieving documents, which is called retrieval-augmented generation (RAG), the answer is often checked by a second model, an "LLM judge," that reads the same documents and decides whether the answer is supported. Both models suffer from the same problem. Retrieved text is long and noisy, and models miss facts buried in the middle of it. Context compressors, small models that rewrite retrieved documents into a short, dense note, help answer-generating models a great deal. This proposal argues that the compressors we have are trained to keep the wrong information for judging. A reader only needs the facts that lead to the right answer. A verifier needs the details a wrong answer would contradict: the exact number, the qualifier, the negation, the second entity with a similar name. A compressor tuned for readers deletes exactly these.
We propose to train a compressor whose reward is a frozen judge's ability, on the compressed evidence alone, to tell a supported answer from small corruptions of it. The compressor does not see the answer being judged. This avoids a circularity in which evidence is picked to match the candidate. It also lets one compressed note be reused across many candidates, prompt variants, and pairwise comparisons. The claim to be tested is specific: at a fixed token budget, compression trained for verification beats both reader-trained compression and candidate-conditioned sentence extraction on hallucination-detection benchmarks. It also loses little or nothing relative to giving the judge the full raw context.
Why this gap is real
The evidence that judges degrade on long references is direct. FaithBench builds hard summarization cases from ten modern models. It finds that automated detectors sit around 50–62% balanced accuracy and are "overly optimistic," calling unwanted hallucinations consistent. Less is More for Long Document Summary Evaluation by LLMs shows that shortening the source before judging can help. Its method, called Extract-then-Evaluate, extracts source sentences that score well on ROUGE, BERTScore, or entailment against the candidate summary, which is the closest existing work. Our proposal differs from it in three ways. It learns what to keep rather than using a fixed heuristic. It is blind to the candidate, whereas Extract-then-Evaluate selects sentences that overlap with the candidate, and a hallucination that shares words with the source can attract the wrong evidence. Finally, it targets multi-document retrieved context rather than a single long document.
The compressors that supply our method, Oreo and BRIEF, reach roughly 13x and up to 29x compression. They are trained with reinforcement from a downstream reader's QA accuracy or from multi-hop synthetic data, and they are evaluated only on answer accuracy. Neither has been tested as input to a judge. EviNote-RAG does connect denoising and judging. However, it uses an entailment judge only as a reward for the reader's notes, and that reward requires the gold answer.
On the constraint side, Is Summary Useful or Not? found that humans answering detail-oriented questions from summaries dropped from 0.32 to 0.22 exact match. That result is a direct warning that compression destroys verification-relevant detail. We treat it as the central risk to measure, not something to argue away. Judge-side methods such as ConsJudge never touch the context: it trains a judge for consistency across k prompts, still feeding the raw documents k times. These methods are complementary consumers of our compressed note.
Core Claim
What a judge needs from evidence can be learned from cheap synthetic contrasts, and it differs measurably from what a reader needs. Let $c$ be the raw retrieved context and $q$ the question. Let $z = C_\theta(q, c)$ be the compressor's output. Let $a^+$ be an answer supported by $c$, and let $a^-_j$ be minimal corruptions of it, such as entity swaps, number and date changes, dropped negations, and invented bridge facts. Let $s_J(z, a)$ be a frozen judge's log-odds that $a$ is supported by $z$. The compressor is trained to maximize
$$R(z) = \frac{1}{m}\sum_{j=1}^{m}\big[s_J(z,a^+) - s_J(z,a^-_j)\big] \;-\; \lambda\,|z| \;-\; \mu\,\#\{\text{propositions in } z \text{ not entailed by } c\}.$$
Here $m$ is the number of corruptions, $|z|$ is the token length, and $\lambda,\mu$ are penalty weights. The first term rewards keeping whatever lets the judge separate truth from near-truth. The second term prices length. The third term uses an entailment model to punish the compressor for injecting its own unsupported claims, which is the second known failure of abstractive compressors.
Research Program
We first build the training contrasts. We draw from HotpotQA, MuSiQue, and 2WikiMultiHopQA, where gold supporting facts span several retrieved passages. For each question we pair the gold answer, and a long-form answer generated with access to the supporting facts, with $m$ corruptions produced by an instruction-tuned LLM. Each corruption is checked by an entailment model to confirm it is actually contradicted or unsupported. Retrieved distractor passages stay in $c$. The training data contains no labeled examples from any evaluation benchmark.
The compressor is a 1–3B instruction-tuned model, chosen to avoid BRIEF's 512-token input ceiling. Its output has two kinds of content: verbatim source spans, and short bridge propositions explicitly marked as non-verbatim. The marking lets us audit how much abstraction is used and lets the judge weigh the two kinds differently. We train it with supervised warm-start on reader-style notes, then Group Relative Policy Optimization against $R(z)$, with the judge frozen. We train against a mid-sized open judge, and then test transfer, without retraining, to other judges, including a frontier API model and FaithLens. If a note trained against one judge only helps that judge, the method is a judge-specific hack. Transfer to other judges is therefore part of the claim.
The analysis component asks what the learned compressor keeps that reader compressors drop. We tag retained source tokens by type (numbers, dates, negations, modal qualifiers, secondary entities) and compare retention rates across compressors at matched length. A secondary question comes from the tension FaithBench reports, that chain-of-thought judging underperformed zero-shot. We test whether that reversal disappears when the judge reasons over the compressed note instead of raw context. If it does, part of the chain-of-thought failure is a noise effect rather than an intrinsic one.
Baselines
The naive reference point is the same frozen judge reading the full raw retrieved context, with identical prompts. This measures whether any compression costs or buys accuracy. Alongside it we run length-matched random truncation, so that gains from simply being shorter are separated from gains from choosing well.
The strongest compression competitor is Extract-then-Evaluate from Less is More, run in its candidate-conditioned ROUGE and entailment variants. Its reported correlations with human judgments are not available: no reported result found on Lacuna for Extract-then-Evaluate on these metrics. It is the strongest competitor because it keeps exact source wording, cannot hallucinate, and sees the candidate, which our method deliberately does not. We run it at the same token budgets (roughly 2x, 4x, 8x, and 16x compression), with the same judge and prompts. We also give it its own tuned selection threshold on a shared development split.
The reader-trained compressor baseline is Oreo and BRIEF style compression, retrained on our base model with the same data and RL compute, but rewarded by reader QA accuracy instead of $R(z)$. This is the ablation that isolates the key ingredient: same architecture, same output format, same budget, and only the reward differs. A second ablation restricts our compressor to verbatim spans only, which tests whether abstraction contributes anything beyond the verification reward.
The strongest existing detectors set the absolute bar. On RAGTruth, a fine-tuned Llama-2-13B reaches 78.7% response-level F1, versus 63.4% for prompted GPT-4 (RAGTruth). On LLM-AggreFact and HoVer, FaithLens reports 86.4% average macro-F1, above GPT-4.1's 83.0%. On FaithBench, FaithJudge with o3-mini-high reports 84% balanced accuracy and 82.1% F1-macro (Benchmarking LLM Faithfulness in RAG), against 62.31% for RAGAS with GPT-4o.
Beating the comparison has a specific meaning for each competitor. Against the raw-context baseline, our compressor must match or exceed it at 4x or greater compression for the same judge. Against Extract-then-Evaluate and the reader-trained compressor, it must win by a statistically reliable margin at every matched budget, with bootstrap intervals over examples. Against FaithLens, the bar is plug-in improvement: FaithLens on our note must score above FaithLens on raw context.
Why not just…?
Why not just extract candidate-relevant sentences instead of training a compressor? This is the most serious threat, and the verbatim-only ablation is designed to settle it. If extractive selection trained with $R(z)$ matches the full compressor, we will report that abstraction is unnecessary and that the contribution is the verification reward. Heuristic, candidate-conditioned extraction has a weakness we can probe directly. When a hallucinated claim shares vocabulary with an unrelated source sentence, lexical selection retrieves that sentence and can starve the judge of the contradicting one. We will report accuracy separately on corruptions constructed to have high lexical overlap with distractor passages. BRIEF's finding that isolated extraction fails to bridge multi-hop gaps (19.19x versus RECOMP's 10.02x compression on HotpotQA, with +3.00 EM) suggests the bridge propositions matter most on multi-hop evidence. That is a reader result, though, and it may not carry over to judging.
Why not just give a long-context frontier judge the raw documents? FaithBench shows frontier detectors remain near chance and overly optimistic on hard cases. Lost-in-the-middle effects (Lost in the Middle) mean a large window does not guarantee attention to every part of it. Cost compounds when judging is repeated across k prompt paths or many pairwise comparisons. A candidate-blind note is computed once and reused, so we will report accuracy alongside judge input tokens per evaluated candidate. If a frontier judge on raw context still beats every compressed variant, the honest conclusion is that compression is a cost tool, not an accuracy tool, and we will say so.
Why not just fine-tune a better judge, as FaithLens does? A strong specialized detector might absorb the noise itself. We therefore treat FaithLens as a downstream judge rather than a rival. If its accuracy on raw context already saturates, and our note gives no gain, the compressor matters only for weak or general-purpose judges. That narrower result is still useful but much less interesting.
Why not just let the judge extract evidence in its own chain of thought, or correct its errors statistically afterward? FaithBench reports chain-of-thought underperforming zero-shot because the reasoning introduces its own hallucinations. The Silent Judge shows rationales that never acknowledge the shortcuts actually driving verdicts. We include a judge-self-extraction prompt as an extra baseline. Post-hoc correction with PPI++ (Efficient Inference for Noisy LLM-as-a-Judge Evaluation) fixes aggregate win-rate estimates, not individual verdicts. The same paper notes its efficiency falls when errors depend on instance features. Missing a buried contradiction is exactly such an error. The two approaches are complementary: we will report PPI++ interval widths with raw-context judges and with compressed-context judges, expecting narrower intervals from the more accurate judge.
Decisive Experiments
We evaluate on RAGTruth (QA subset with MS MARCO passages, plus summarization), FaithBench, and the RAG and multi-hop portions of LLM-AggreFact and HoVer. Metrics are response-level F1, balanced accuracy, and macro-F1, reported per hallucination category. On FaithBench we report the "unwanted" category separately, since subtle errors are where lossiness should bite.
Three outcomes decide the proposal. First, the motivating premise must hold. Reader-trained compression must lower judge accuracy relative to raw context on hallucinated examples at matched budgets, while leaving reader QA accuracy intact. If it does not, the reader–verifier distinction is not real and the project narrows to cost reduction.
Second, the verification-trained compressor must beat the reader-trained one under the identical-architecture ablation. It must also beat tuned Extract-then-Evaluate at every budget, and it must be no worse than raw context at 4x compression or greater, for at least two judges it was not trained against.
Third, the retention analysis must show that the gain comes from keeping checkable details such as numbers, negations, and secondary entities. The faithfulness penalty must also keep unsupported propositions in the note rare, as measured by entailment checks and a small human audit. A win on accuracy produced by a compressor that invents bridge facts would not count.
References
- FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs
- Less is More for Long Document Summary Evaluation by LLMs
- Oreo: A Plug-in Context Reconstructor to Enhance Retrieval-Augmented Generation
- BRIEF: Bridging Retrieval and Inference for Multi-hop Reasoning via Compression
- EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notes
- Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Tasks
- Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models
- RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
- FaithLens: Detecting and Explaining Faithfulness Hallucination
- Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards
- Lost in the Middle: How Language Models Use Long Contexts
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Efficient Inference for Noisy LLM-as-a-Judge Evaluation