Long-context language models keep a memory of every past token, the key-value (KV) cache, and this memory now dominates serving cost. Holding the full cache for LLaMA-3.1-8B at one million tokens takes more than 120GB of GPU memory (OBCache). One effective way to shrink it without retraining is to decide, attention head by attention head, which heads get to see the whole context. RazorAttention keeps the full cache only for the roughly 15% of "retrieval heads" that copy or match earlier tokens. Every other head gets a short recent window. A separate line of interpretability work shows that the tendency of models to be derailed by irrelevant or false context comes from a small set of 3–10% of heads. These heads reflexively boost the logit of whatever token appears in the prompt (Llama See, Llama Do). That copying behavior is close to what makes a head look like a retrieval head. The compression method may therefore be spending its full-memory budget on exactly the heads that make the model gullible.
We propose to choose which heads keep long-range memory using two cheap, forward-only scores per head. The first measures how much the head is needed for real long-range recall. The second measures how much it pushes the model toward distractor tokens. Heads that are distractible but not needed for recall are restricted to a short local window instead of being zeroed out. The central bet is mechanistic. A head can only entrain on a distant distractor if it can attend to it, so withdrawing a head's long-range cache should remove its distracting influence while keeping its local function. If the bet holds, one change to the cache policy gives a smaller cache, higher decoding throughput, and better accuracy on noisy prompts, all without gradients.
Core Claim
The claim has three parts, and each can be tested. First, the heads selected for full caching by retrieval-head criteria overlap substantially with the heads that drive contextual entrainment. Nobody has measured this overlap; it is Gap 1 in our file. Second, giving entrainment-dominant heads only a local cache, rather than ablating them, suppresses distraction when the distractor is far from the generation point. It should also avoid the "Hydra effect" of backup circuits and the small capability loss that zeroing causes. Third, both scores can be computed with forward passes on a few hundred contrastive prompts. That keeps the method training-free and puts its calibration cost in the same class as the per-layer sensitivity search of ASVD. This differs from Llama See, Llama Do, which finds the circuit by gradient-based differentiable masking and intervenes by deletion. It also differs from RazorAttention, whose head selection ignores robustness entirely.
Research Program
The first stage is the overlap study on Llama-3-8B, where the entrainment circuit is already published, and on Qwen2.5-7B. For each head $h$ we compute a retrieval score $R_h$ using RazorAttention's echo and induction criteria on needle-style prompts. We also compute an entrainment score using direct logit attribution: we project the head's output at the final position onto the vocabulary and compare the distractor token against the correct answer,
$$E_h = \mathbb{E}_{(x,a,d)}\big[\langle W_U^{\top} o_h(x),\; e_d - e_a\rangle\big],$$
where $x$ is a prompt containing a counterfactual or irrelevant statement, $a$ is the correct answer token, $d$ is the distractor token, $o_h(x)$ is head $h$'s contribution to the residual stream after its output projection, $W_U$ is the unembedding matrix, and $e_d, e_a$ are one-hot vectors. This needs only forward passes. We validate $E_h$ by checking that its top heads recover the masking-derived circuit of Llama See, Llama Do, and we report the overlap as a rank correlation.
The second stage is the policy itself. A head keeps the full cache if it is in the top retrieval set and its $E_h$ falls below a threshold $\tau$. All other heads keep attention sinks, a recent window, and RazorAttention's mean compensation token. This preserves the head-level structure that keeps FlashAttention usable. The threshold $\tau$ and the window length are tuned on a held-out calibration split. The calibration distractors are simple counterfactual statements in the style of the relation-extraction data. The evaluation distractors come from different generators, so the method cannot win by memorizing a distractor style.
The third stage tests the mechanism directly. We vary the distance between the distractor and the question, from inside the local window to tens of thousands of tokens away. We expect the robustness gain to be large for distant distractors and to vanish for adjacent ones. If gains appear regardless of distance, the local-window explanation is wrong and something else is going on. Finally, we measure Time-to-First-Token, inter-token latency, peak memory, and decoding throughput on GPU, since RazorAttention's throughput figure was reported on Ascend NPUs.
Baselines
The naive reference point is the uncompressed model with full KV cache and no intervention. It sets the accuracy ceiling on clean inputs, the latency and memory floor, and the amount of damage distraction does with no mitigation.
The strongest existing competitor is RazorAttention. It is the only training-free, head-level cache method in the file with long-context evidence, reporting 70% cache reduction, near-100% Needle In A Haystack accuracy at 1M tokens, and a 71.8% decoding throughput gain at 256K context on 8 Ascend 910B NPUs. For RULER and LongBench we found no reported task-level number on Lacuna for RazorAttention, so we will rerun it ourselves. Our policy differs from it only in head selection, so we will match the compensation token and window size. We will match total cache memory exactly, by moving freed full-cache slots to the next-best retrieval heads when needed, and both methods get the same calibration prompt count. Beating it means equal clean-task accuracy within noise, equal memory, and a significant gain on distractor-injected tasks. If we also gain throughput by keeping fewer full-cache heads, that counts as a second win.
The token-level competitors are OBCache, which uses second-order output-aware saliency and is reported to keep high accuracy on LLaMA-3.1-8B RULER-4K while discarding 70–90% of the cache (no exact per-task number on Lacuna), and SnapKV-style eviction. Both run at matched cache ratios.
The main ablation isolates our key ingredient by adding entrainment-head zeroing on top of RazorAttention. Zeroing is the intervention from Llama See, Llama Do, which reports MMLU going from 56.3% to 55.5% and a distractor's logit rank falling from 2nd to 53rd on country-capital extraction. A second ablation uses random head assignment at the same budget. A third uses masking-derived scores in place of $E_h$, to check whether forward-only scoring loses anything.
Why not just…?
Why not just run RazorAttention unchanged? If the overlap study finds that retrieval heads and entrainment heads are disjoint, RazorAttention is already robust by accident, and our contribution shrinks to a negative finding. That finding is still useful because it settles Gap 1. If they do overlap, RazorAttention gives full memory to distracting heads, and the distractor-distance experiment will show whether that costs accuracy.
Why not just zero the entrainment heads on top of any compressor? This is the most serious alternative. Llama See, Llama Do itself reports two problems with it: backup heads re-establishing distraction, and a measurable capability loss. Zeroing also removes heads that may be needed for recall, if overlap is high. Our ablation compares zeroing against local-window restriction head for head. If zeroing matches us on both robustness and needle recall, we will report that the simpler intervention suffices and that the remaining value of our work lies in the forward-only scoring.
Why not just use query-aware token selection, which drops irrelevant tokens anyway? ProphetKV retains 96–101% of full-prefill accuracy at a 20% recomputation ratio. However, it selects the tokens the query attends to, and a semantically coherent distractor is precisely a token the query attends to. ProphetKV also assumes retrieval is not mostly noise. In addition, SCBench shows that permanently dropping tokens collapses multi-turn accuracy. Our method keeps every token available to the full-cache heads, so we include SCBench multi-turn runs where token dropping should fail and we should not.
Why not just tell the model to ignore irrelevant information, or fine-tune it? Breaking Focus reports that "ignore irrelevant info" prompting is ineffective, while preference fine-tuning (DPO) helps. We include the prompting baseline as a cheap control. DPO requires training, so it falls outside the training-free regime. We report it as an upper reference where compute allows.
Proposed Experiments
Clean-capability checks use MMLU, Needle In A Haystack up to 128K, RULER from 4K to 128K, LongBench, InfiniteBench, and SCBench multi-turn mode, all at matched cache ratios of 50%, 70%, and 85%. RULER's distractor-QA and multi-hop tracing tasks are the most relevant here.
Noisy-context checks inject distractors at controlled distances into RULER and LongBench QA. They also cover NaturalQuestions-Open and SQuAD with 10 to 100 retrieved documents, following the setup where Improving Zero-shot Reader shows accuracy falling as documents increase, and MuSiQue with its constructed distractor passages (MuSiQue). Finally, we apply adaptive tree-search distractions on CommonsenseQA, where Breaking Focus reports GPT-4o-mini dropping from 85.7% to 22.0%. These are run on open models, since we need the attention heads.
Systems measurements cover decode throughput, Time-to-First-Token, inter-token latency, and peak memory on a single GPU at 32K and 128K. They also include the one-time calibration cost compared with ASVD's sensitivity search.
There are three decisive checks. The first is whether the measured overlap between retrieval and entrainment heads is substantial. The second is whether local-window restriction beats zeroing on needle recall at equal robustness. The third is whether the robustness gain grows with distractor distance. If the third check comes out flat, the proposed mechanism is falsified even if accuracy improves, and we will say so.
References
- OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
- Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs
- ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
- ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition