Does Compressing Retrieved Context Make RAG Answers Less Grounded?

Retrieval-augmented generation (RAG) means fetching passages from a document collection and pasting them into a language model's prompt so that its answers rest on sources rather than on memory. The pasted passages are long and expensive to process, so a recent line of work compresses them, keeping only a few percent of the tokens. Some of these compressors even raise answer accuracy. We suspect the accuracy gain hides a cost. A compressor rewarded only for correct answers can learn to keep a few topical words that cue the model's memory and discard the sentence that actually states the fact. The answer is still right and still cites a passage, but the passage no longer caused the answer. This is the "post-rationalization" failure that Correctness is not Faithfulness in RAG Attributions documented in uncompressed RAG. Nobody has measured whether compression makes it worse.

We propose a cheap test of whether an answer depends on its retained context. The test edits the fact in the source, compresses the edited text, and checks whether the answer follows the edit. We then use the same test as a training reward for a context compressor. Two claims follow, and each can be checked. First, compressors tuned on exact-match accuracy lower this dependence rate, and they raise the error rate on questions where the model's memory is wrong, even while their exact-match scores go up. Second, a compressor trained to preserve dependence gets similar token savings and similar accuracy without that loss. Metric misalignment is the lens here, compression cost is the constraint, and hallucination on questions that conflict with the model's memory is where the damage shows.

Core Claim

Exact match (EM) is a proxy, and when compressors are optimized against it, the proxy drifts away from the property users care about: the answer is true because of the cited evidence. The effect should be largest where memory and evidence disagree. That includes long-tail entities, counterfactual premises, and facts that have changed. On ordinary benchmark questions memory usually agrees with Wikipedia, so EM cannot see the problem.

We define the counterfactual reliance rate. Take a question with a gold evidence sentence and swap the answer span for a plausible alternative of the same type. Run the full pipeline, compressor included, on the edited documents. The answer counts as reliant if it is correct on the original documents and switches to the substituted value on the edited ones. The metric only needs text edits made before compression. So it works on hard (text-pruning) compressors and also on soft compressors that turn text into vectors, such as OSCAR. Soft compressors are the case where text-based audits like FACTUM cannot be computed at all.

Relation to Prior Work

Two papers motivate the proposal by showing that the gap exists. Wallat et al. injected answer snippets into documents and found that Command-R+ cited relevant documents it had not actually used 57% of the time. They showed that citation correctness is not citation faithfulness, but they offered no fix and did not study compression. MIRAGE split RAG failures into four categories: noise vulnerability, context acceptability, context insensitivity (ignoring gold context in favor of memory), and context misinterpretation. It measured these only for uncompressed retrieval. Its context-insensitivity metric is the aggregate symptom our per-example test targets, and its question pool includes the counterfactual IfQA questions and long-tail PopQA questions we need.

The compressors under study include Less Is More (CORE-RAG). It trains a text compressor with Group Relative Policy Optimization (GRPO, a reinforcement-learning method that compares sampled outputs within a group), using EM/F1 as the reward. The authors themselves flag the risk that the compressor learns shortcuts. ECoRAG ranks sentences by how directly they support the answer and uses a small evaluator to decide whether enough has been kept. We borrow its sentence-level output format. We replace its answerability labels with a signal that directly measures whether the answer depends on the kept sentences. ACC-RAG and OSCAR compress into vectors. We use them only as audit targets.

On the detection side, FACTUM reads model internals at the position where a citation token is emitted. RLSeek trains a detector to quote evidence before it labels hallucinated spans. Both check a generation after it is produced. Our method acts earlier, on what the compressor keeps, and it needs neither white-box access nor span labels.

Two papers constrain the design. On A Scale From 1 to 5 shows that LLM judges carry model-specific strictness and leniency biases, which rules out using an LLM judge as the reward. Rowen warns that consistency-based signals can misfire when errors are shared across the perturbations. Our perturbation is an edit whose effect we know exactly, so it avoids that problem. Its cost is one extra forward pass per training rollout. The deployed system pays nothing extra.

Research Program

The first stage is a measurement study. We take off-the-shelf compressors and our own re-implementation of the CORE-RAG recipe and run them in front of Llama-3.1-8B and one larger open generator. We report EM, token savings, the MIRAGE diagnostics, and counterfactual reliance, broken down by question type: head-entity NQ and TriviaQA, long-tail PopQA, and counterfactual IfQA.

We will build counterfactual edits automatically. An edit swaps the answer span in the gold chunk for a same-type entity or number, and the surrounding text is checked for consistency. A human audit on a sample will confirm that edits are fluent and do not give themselves away.

We will validate the metric against a known ground truth in two ways. The first reuses Wallat's injection protocol, where we know which document contains the answer. The second is a small human study. Annotators see the kept context and the answer and judge whether the answer is supported by that context specifically. This is the step that connects to metric–human alignment: we report how well EM, citation correctness judged by natural-language inference (NLI, a model that decides whether one text supports another), and counterfactual reliance each agree with the human faithfulness judgments.

The second stage trains a compressor with a reliance reward. We keep the architecture, data, and GRPO budget identical to the CORE-RAG replication and change only the reward. For each rollout, the compressor keeps a set of sentences from the original documents. The generator answers from those sentences. Separately, the compressor is run on the edited documents and the generator answers from its output. The reward is

$$r = \mathrm{EM}(\hat a, a) + \lambda\,\mathrm{EM}(\hat a', a') - \beta\,\frac{|c|}{|D|},$$

where $\hat a$ and $\hat a'$ are the answers produced from the original and edited documents, $a$ and $a'$ are the true and substituted answers, $|c|/|D|$ is the fraction of tokens kept, and $\lambda, \beta$ are weights chosen on a development set.

Distractor chunks stay in the input, as in MIRAGE's Mixed setting, and only the gold sentence is edited. This guards against an obvious failure: a compressor that learns to trust any entity it sees. We measure whether that failure occurs anyway through MIRAGE noise vulnerability.

Baselines

The naive reference points come first. One is the generator with no retrieval, which measures memory alone. Another is full-context RAG with fixed top-$k$ at $k=1$, $3$, and $5$. MIRAGE reports that going from top-3 to top-5 hurt GPT-3.5-Turbo, so small $k$ is a real competitor for cheapness.

The strongest existing method is CORE-RAG. The file reports a 3.3-point EM gain on NQ over full-context RAG while keeping about 3% of tokens. We re-run it ourselves on identical retrieval pools, with the same generator, the same compressor size, and the same number of GRPO steps as our method, so the only difference is the reward.

ECoRAG is the strongest competitor that prunes at the sentence level. Its paper reports 65.34 EM on TriviaQA versus 56.21 for standard RAG, using about 600 tokens instead of about 14,000. We run the released recipe with our generator.

OSCAR and ACC-RAG serve as soft-compression references. For OSCAR, the file reports accuracy matching no compression on NQ. For ACC-RAG, it reports a match score of 41.11 against 44.49 for vanilla RAG on NQ. Both are audited with our reliance test but not retrained.

The key ablation adds an NLI sufficiency term to EM in place of our counterfactual term, under the same budget. This separates "the kept text supports the answer" from "the answer depends on the kept text."

We beat the baselines if two things happen at matched token budgets, within ±1 point of the percentage of tokens kept. Our compressor must stay within 1 EM point of CORE-RAG on NQ and TriviaQA, while significantly raising counterfactual reliance and IfQA/PopQA accuracy and lowering MIRAGE context insensitivity. It must also do this without raising noise vulnerability.

Why not just…?

Why not just retrieve fewer documents? Small top-$k$ is cheap, and it avoids learned compression entirely, so it cannot learn shortcuts. It is a real threat for the faithfulness question specifically. MIRAGE's top-$k$ results and ECoRAG's gap over standard RAG show that whole-document retrieval either misses evidence or pulls in distractors, but neither paper measured reliance. We include top-1 and top-3 at matched token counts. If they match our compressor on reliance and memory-conflict accuracy, the practical conclusion is that learned compression is not worth its risk, and we will report that.

Why not just add an NLI or citation-correctness check to an EM-trained compressor? Entailment tells you the kept text supports the answer, not that the model used it. Wallat et al. found high citation rates for documents that were never causally used, and those citations would pass an entailment check. The NLI-sufficiency ablation tests this directly. If it matches our reward on reliance, the counterfactual edit is unnecessary.

Why not just use an LLM judge or a trained hallucination detector as the reward? LLM judges are biased in ways that depend on the model (Jing et al.). They also underperform fine-tuned detectors on RAGTruth: 63.4% versus 78.7% response-level F1 in RAGTruth. Running a judge on every rollout would also dominate training cost. A fine-tuned detector is a fairer challenger, and we use the RAGTruth-style detector as an evaluation-only check on open-ended outputs. The deeper problem is that a detector trained on correctness labels shares the blind spot described in the previous answer.

Why not just probe the model's internals, as FACTUM does? Probing needs white-box access to activations and cannot be applied to soft-compressed context. Our test applies to any pipeline. We will report FACTUM scores on the white-box generator as a second, independent faithfulness signal.

Proposed Experiments

The decisive first experiment tests whether EM and reliance diverge. On MIRAGE's pool, we plot EM and counterfactual reliance against the fraction of tokens kept for each compressor. The hypothesis fails if reliance stays flat while EM rises.

The second experiment tests whether the divergence shows up on memory-conflict questions. We compare IfQA and long-tail PopQA accuracy, together with MIRAGE context insensitivity, between compressed and full-context pipelines.

The third experiment tests whether the reliance reward closes the gap at the same cost. We compare our compressor against the CORE-RAG replication and the NLI ablation. We report EM on NQ and TriviaQA, reliance, noise vulnerability, prefill tokens, time to first token, and total GRPO GPU-hours, including the roughly doubled number of rollout passes.

For open-ended generalization, we apply the trained compressor to RAGTruth-style generation. We score hallucinated spans with a fine-tuned detector and a small human sample, and report which automated metric best tracks the human judgments.

Finally, we audit OSCAR and ACC-RAG with the reliance test. This checks whether vector compressors, which no text-based audit can examine, silently shift the model toward answering from memory.