Retrieval-augmented generation (RAG) means handing a language model a stack of retrieved passages to read before it answers. Those passages are long and mostly irrelevant, so systems prune them. Today's pruners all pay for that with a second model: a trained tagger, a 7B scorer, or an attention pass run after the question arrives. That extra pass costs latency. Worse, because the pruned text depends on the question, the model cannot reuse its stored internal state for each document, the key-value cache (KV cache: the per-token keys and values a transformer saves so it need not reprocess text it has already read). This proposal tests a cheaper route. When the reader model first processes a document offline to build that cache, it already produces a hidden vector for every token. The length (Euclidean norm) of those vectors is known to track how much information a token carries. We propose to record those norms at cache-building time for free. At question time, a lightweight norm-weighted match between the question and the cached states decides which sentences' cache entries to drop. Nothing is re-encoded and no auxiliary model is run.
The claim is only interesting if the reader's contextual norms carry information that cheap corpus statistics do not. Input-embedding norms largely re-encode word frequency, so pruning by them might be nothing more than inverse document frequency (IDF, the classic rare-word weight) in disguise. The central experiment is therefore designed to fail loudly if contextual norms are no better than IDF.
Core Claim
In a RAG reader that serves documents from a precomputed KV cache, the $L_2$ norms of intermediate hidden states, taken from the reader itself, give a token significance signal that has three properties. It costs nothing, because the norms are a byproduct of the prefill (the offline pass that builds the cache). It is contextual, because the same word gets a different norm depending on its role in its passage. And combined with a relaxed, norm-weighted transport match against the question, it supports sentence-level eviction of cached entries. We predict this eviction matches trained or auxiliary-model pruners at comparable compression while keeping time-to-first-token close to that of an unpruned cached system.
Why This Is Different From What Exists
Norm-based significance has a clear record outside generative RAG.
The papers that supply the method show where the signal comes from:
- C-PHRASE and Neural Vector Spaces for Unsupervised Information Retrieval show that training makes discriminative words grow longer vectors.
- Word Rotator's Distance separates a vector's norm (used as probability mass) from its direction (used as cost) for similarity matching. Its STS-B Pearson score rose to 77.56 from 65.74 for standard Word Mover's Distance on GloVe. State Mover's Distance applies the same move to encoder states in translation.
- Semantic Kernels picks the highest-norm source tokens as anchors for a translation decoder.
- NormXLogit multiplies input-embedding norms by logit attributions to explain classifiers.
None of these prune retrieved evidence for a generator.
The closest existing work is Contextualized Query Embeddings for Conversational Search. It observes that a BERT query encoder's token norms act as term weights and extracts high-norm tokens as a sparse query. That use is on the query side of retrieval, with a heuristic threshold. We apply the observation to the document side of generation, inside the reader's own cache.
On the RAG side, the pruners motivate the gap by showing how much pruning helps, and they also show its cost:
- Provence trains a DeBERTa-v3 cross-encoder to tag tokens to keep.
- LongLLMLingua scores context with a small LLM's contrastive perplexity. It explicitly notes that question-dependent rewriting prevents cached document states.
- AttentionRAG runs an extra attention pass toward a focal token.
- Token Elimination drops tokens during Fusion-in-Decoder decoding using cross-attention.
- PISCO trains soft memory tokens.
Every one of these either needs a separate trained component or recomputes the context per question. Our design needs neither.
Two findings constrain the design:
- A Study on Token Pruning for ColBERT reports a "repetition trap": top-scoring tokens are often duplicates or variants of each other. We therefore select sentences under an explicit redundancy penalty rather than taking top-norm tokens.
- NormXLogit reports that norm scores identify tokens a prediction needs (comprehensiveness) better than tokens that alone preserve it (sufficiency). Pruning is a sufficiency problem. We answer this in two ways. We prune at sentence granularity. And we drop cache entries after the prefill has already mixed information across tokens, so kept tokens partly carry what dropped tokens said. Whether that answer holds is an empirical question, and the experiments measure it directly.
Research Program
Offline scoring. For each corpus passage, the reader model runs its normal prefill and stores the KV cache. For each token $i$, it also stores one scalar $s_i = \|\tilde h_i^{(\ell)}\|_2$. Here $h_i^{(\ell)}$ is the hidden state at layer $\ell$, and $\tilde h$ is that state with its few persistently extreme outlier dimensions masked. Those dimensions produce huge norms on delimiter and first-position tokens regardless of content. The dimensions are identified once per model, on held-out text. We also cache a low-dimensional projection of each $h_i^{(\ell)}$ for matching. Normalizing the norms within a passage gives a mass $m_i = s_i / \sum_k s_k$. Both the layer $\ell$ and the outlier mask are chosen on a development split and then frozen.
Query-time selection. The question is encoded by the same reader up to layer $\ell$, giving states $q_1,\dots,q_J$. Each cached sentence $S$ is scored with a relaxed form of Word Rotator's Distance, in which each document token sends its mass to its closest question token:
$$\mathrm{score}(S) = \sum_{i \in S} m_i \,\max_{j} \cos(h_i, q_j)$$
Using the relaxed form avoids the cubic cost of full optimal transport that the file flags for these methods. Sentences are then chosen greedily until a token budget is met. After the first pick, each candidate's score is discounted by its cosine similarity to the norm-weighted centroid of the sentences already kept. This penalty addresses the repetition trap and the cross-passage redundancy that Provence leaves open. The reader then decodes over the cached keys and values of the selected sentences only, keeping their original positions. Nothing is recomputed.
Analysis. We treat the mechanism, not just the headline score, as the object of study. Three questions matter:
- How much of the variance in contextual norms is explained by IDF and by input-embedding norms, and does the remainder predict answer-bearing sentences? This is the residual test.
- Does the norm signal behave differently for rare versus common terms? A White Box Analysis of ColBERT found that rare terms stay stable across contexts while common terms absorb context, which predicts that the contextual signal adds most on common words.
- How much does answer accuracy degrade from dropping entries after mixing, compared with re-encoding the same kept sentences from scratch? The gap between these two conditions measures the sufficiency cost directly.
Baselines
The naive reference points are the full, unpruned retrieved context served from cache and first-$k$ truncation to the same token budget. First-$k$ is known to be deceptively strong on front-loaded data such as MS MARCO. A Study on Token Pruning for ColBERT reports First-$k$ at MRR@10 0.357 versus a 0.365 unpruned baseline at 50 tokens. A method that cannot beat truncation at equal budget has no case.
IDF sentence scoring. The first ablation replaces $m_i$ in the scoring equation with normalized IDF and keeps everything else fixed. It is the most dangerous competitor because it is free, needs no model, and also preserves caches.
Input-embedding norms. The second ablation uses input-embedding norms in place of contextual ones, in the spirit of NormXLogit's embedding factor. It isolates exactly the question left open in the file: whether static norms suffice or contextual norms are needed.
Uniform mass. A third ablation sets $m_i$ uniform while keeping the question-matching step. It tests whether the norms matter at all, or whether the matching does all the work.
Strongest trained pruner: Provence. Provence is the strongest competitor for quality, because it is trained specifically to tag context tokens for RAG readers. The file reports only a qualitative result for it: pruning improved PopQA accuracy by removing distractors. No reported numerical result was found on Lacuna for Provence on NQ or PopQA, so we will run the released model ourselves.
Strongest auxiliary-model pruner: LongLLMLingua. The file reports that it improved GPT-3.5-Turbo on NaturalQuestions by 21.4% while using 25% of the original tokens. That figure is relative to a different reader and uncompressed setting, so it is context only, not a target.
Attention-based selection: AttentionRAG. Its reported TriviaQA exact match of 0.89 against 0.87 uncompressed, with a 70B reader, shows that pruning can beat the full context.
Fair comparison. All methods use the same retriever and top-20 passages, the same reader (an 8B-class open model, with a 70B check on one dataset), and the same token budgets of 25%, 12.5%, and 6%. Each method gets the same development-set tuning budget for its hyperparameters: our layer and penalty, Provence's threshold, and LongLLMLingua's ratio settings. Latency is measured end to end on the same hardware and counts every forward pass each method needs after the question arrives. Beating a baseline means matching Provence's accuracy within one point at equal compression while cutting time-to-first-token substantially. It also means beating IDF and input-embedding norms by a margin larger than seed-to-seed variation on at least three of the four QA datasets.
Why not just…?
Why not just weight sentences by IDF? This is the real threat, and it is the reason the design centers on contextual rather than static norms. The file's evidence against IDF is indirect. In the ColBERT pruning study, Top-$k$ IDF lost to learned token selection under aggressive compression. Contextualized Query Embeddings shows contextual norms suppressing and amplifying terms by conversational role, which static counts cannot do. But neither result is about generative RAG. The IDF-mass ablation settles the question. If it matches contextual norms, we will report that norm scoring adds nothing here, and the contribution reduces to the cache-preserving selection procedure.
Why not just use attention weights, which the prefill also produces? Attention received per token is a natural competing free signal. There are three reasons it may not be as good. First, optimized attention kernels do not store the attention matrix, so capturing it costs memory, while a norm is one scalar per token. Second, the ColBERT pruning study found that attention-score pruning falls into the repetition trap. Third, NormXLogit reports that attention rollout was slower and less faithful than norm-based attribution. We still include attention-mass selection under the same greedy, redundancy-penalized procedure. If it ties, the finding becomes "any free prefill signal suffices," which is still useful.
Why not just run Provence or LongLLMLingua and accept the latency? If quality is all that matters, that may be the right call. These methods re-read the context with a second model for every question, and LongLLMLingua states that this breaks document caching. Our latency measurements quantify the price. Our accuracy measurements show whether the trained methods' quality advantage justifies it. A result where Provence wins by several points at every budget would mean norms are only a cheap fallback, and we would say so.
Proposed Experiments
The main evaluation uses Natural Questions and TriviaQA for single-hop factoid answers. It uses PopQA, where When Not to Trust Language Models found that retrieval misled GPT-3 about 10% of the time, so distractor removal should show up there. It uses 2WikiMultihopQA for evidence spread across documents, where redundancy handling and middle-of-context evidence matter. We report exact match or accuracy against compression ratio, along with time-to-first-token and total generation time.
First decisive check: the residual test. We regress the contextual norms on IDF and input-embedding norms, then test whether the residual alone ranks gold-evidence sentences above chance on a held-out NQ split. If it does not, the core claim fails regardless of downstream scores.
Second decisive check: evict versus re-encode. We compare evicting cache entries after the prefill with re-encoding the same kept sentences, at 6% budget on 2WikiMultihopQA. If eviction is clearly worse, the sufficiency concern from NormXLogit binds, and the method is limited to mild compression.
Third check: the redundancy penalty. We ablate the penalty on multi-hop data, where duplicated evidence across passages is common. This shows whether the repetition trap reported for ColBERT carries over to generation-stage pruning.
References
- Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model
- Neural Vector Spaces for Unsupervised Information Retrieval
- Word Rotator's Distance
- Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mapping
- Guiding Neural Machine Translation with Semantic Kernels
- NormXLogit: The Head-on-Top Never Lies
- Contextualized Query Embeddings for Conversational Search
- Provence: efficient and robust context pruning for retrieval-augmented generation
- LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
- AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation
- Optimizing Retrieval-augmented Reader Models via Token Elimination
- PISCO: Pretty Simple Compression for Retrieval-Augmented Generation
- A Study on Token Pruning for ColBERT
- A White Box Analysis of ColBERT
- When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories