Language models already get worse when a prompt is padded with irrelevant text. Adding plausible distractor passages can cut accuracy roughly in half, even when the model answers correctly once the distractors are removed. Separately, serving long prompts cheaply means compressing the model's cache of past-token keys and values to 2 bits or fewer. The cost of that compression is usually measured on clean benchmarks. We propose that the two problems are linked by a specific, fixable mechanism. Rounding errors in cached keys raise, on average, the attention that the compressed tokens receive. The same prompt tokens are always kept at full precision: the "attention sink" tokens where the model parks attention it wants to spend on nothing, and the most recent tokens. So compression moves attention away from those tokens and onto the long compressed middle of the prompt. In a noisy prompt, that middle is mostly distractors. Aggressive cache compression may therefore turn the model's built-in "ignore this" mechanism into more distraction.
This mechanism also changes how the memory budget should be spent. Low-bit schemes store a high-precision scale factor for every small group of values. This "metadata" can add 0.5 to 1 bit per value. Finer groups reduce rounding error but cost more metadata. We argue that most of the harm from coarse groups on distractor-heavy prompts comes from a systematic shift in attention. That shift can be computed and subtracted using the scale factors the cache already stores, so fixing it costs no extra memory. If this holds, a coarse-group cache plus a free correction should match a finer-group cache at a lower effective bit-width on the prompts where compression matters most.
Core Claim
The claim has three parts, each testable.
- Amplification. Under low-bit key quantization, the accuracy loss caused by distractors grows faster with the number of distractors than it does at full precision.
- Cause. Most of that extra growth comes from a mean upward bias in the attention logits of quantized tokens, paid for by the attention mass on full-precision sinks. It does not come from random reconstruction error.
- Budget. Removing the bias with a correction computed from stored scales lets a coarser grouping, with fewer metadata bits per value, reach the distractor robustness of a finer grouping. The comparison is made at matched effective bits, counting all metadata.
Mechanism and What Is Borrowed
The mechanism comes from Quantized Keys Steal Attention. That paper showed that zero-mean noise on keys, passed through the convex exponential inside softmax, raises the expected attention of quantized tokens. It fixed video diffusion caches at INT2 with a second-order logit correction proportional to $\|q\|^2\Delta^2$, where $q$ is the query and $\Delta$ the quantization step size. We reuse that correction as the method-supplying ingredient. We do not claim the bias itself as new.
We differ from that paper in three ways:
- Setting. That paper studied video generation and LLM chunked prefill. It explicitly called single-token decoding less applicable, and it never looked at distractors, sinks, or metadata allocation.
- Channel-wise form. In caches quantized per channel over groups of tokens, as in KIVI, each group $g$ stores one step size $\Delta_{g,c}$ per channel $c$. Treating rounding error as uniform, with variance $\Delta^2/12$, the expected exponentiated logit of token $i$ is inflated by a factor $e^{b_i}$, with
$$ b_i \approx \frac{1}{2d}\sum_{c=1}^{d} q_c^2\,\frac{\Delta_{g(i),c}^2}{12}, $$
where $d$ is the head dimension and $g(i)$ is the group containing token $i$. Every quantity is either the current query or metadata already in memory. The correction therefore needs no new storage and costs one length-$d$ product per group per decode step.
- Diagnosis. We name what the bias does in noisy prompts. A shift that is uniform across all tokens would cancel in softmax. Here the sinks, protected at full precision as in KVSink, and the recent window get no boost. The total mass transferred to the compressed middle scales roughly as $\sum_{i \in \text{quantized}} a_i(e^{b_i}-1)$, where $a_i$ is the full-precision attention weight. That sum grows with the number of distractors.
A second, separate mechanism can also hurt. The random part of the rounding error spreads the logits of $N$ distractors. The largest of $N$ noisy logits grows roughly like $\sigma\sqrt{2\ln N}$, where $\sigma$ is the logit noise standard deviation (the standard extreme-value result for Gaussian maxima). So some distractor can overtake the needle by chance. Mean correction cannot fix this; only smaller step sizes, and therefore more metadata, can. Separating these two mechanisms is the point of the study.
Motivating and Constraining Work
Several papers motivate the question.
- Breaking Focus generated coherent distractors that dropped GPT-4o-mini on CommonsenseQA from 85.7% to 22.0%, and found that "ignore irrelevant text" instructions do not help.
- Cho et al. showed zero-shot readers degrading as retrieved documents go from 10 to 100.
- RULER showed that effective context falls well short of claimed context on tasks with distractors.
- Yazan et al. is the only work in the file that crosses quantization with retrieval noise. It is limited to 4-bit weight-only quantization and personalization tasks, and finds effects that vary by model. It leaves open whether cache compression specifically interacts with distraction.
Other papers constrain the design.
- KVSink shows sinks must be protected: LLaMA2-70B 4-bit perplexity is 59.5 without proper sink preservation, 18.7 with it, and 16.2 at FP16. We cannot simply quantize sinks to remove the precision imbalance.
- NSNQuant and QJL remove per-block metadata entirely. They are the natural test of whether "no scales" beats "cheap scales plus correction." Neither has been evaluated under distractor load.
- The metadata accounting follows BCQ, where 4-bit values become 4.5 to 4.6 effective bits, and BiE on the block-size trade-off.
Research Program
The work has four stages, all on open models: Llama-3.1-8B and Mistral-7B, which the stealing paper also tested, with Llama-3.1-70B as a scale check.
Stage 1: Build a controlled distraction harness. A single gold passage is embedded among $N \in \{0, 4, 16, 64, 256\}$ distractor passages, and $N$ is varied while the question and gold passage stay fixed. Sources are:
- RULER QA and multi-needle tasks.
- NaturalQuestions with DPR-retrieved documents, following Cho et al.
- CommonsenseQA and MMLU items with tree-searched coherent distractors, regenerated with the Breaking Focus procedure.
Stage 2: Measure amplification. We sweep the key cache over 2 and 3 bits and group sizes of 32, 64 and 128 tokens, with values fixed at a moderate per-token setting. We report accuracy against $N$ and the "amplification gap," defined as quantized accuracy loss minus full-precision accuracy loss at each $N$. We also log per-head attention mass on sinks, on the gold span, and on distractors.
Stage 3: Test the predicted mechanism. We check whether the predicted transfer $\sum a_i(e^{b_i}-1)$ correlates with the measured loss of sink mass across layers and heads. We check whether the amplification gap tracks that transfer or instead tracks the logit noise variance.
Stage 4: Bias correction and budget reallocation. We apply the scale-derived correction at decode time and trace the accuracy–effective-bits frontier for three configurations: corrected coarse groups, uncorrected fine groups, and metadata-free quantizers.
Baselines
Full-precision cache. The 16-bit cache run on the same distractor harness is the reference point. Every amplification gap is measured relative to it, and any method's residual gap is our main number.
Naive per-token quantization. Plain round-to-nearest with per-token scales, sinks and recent window left unprotected, is the naive compressed reference. It shows how much any structure helps at all.
KIVI. KIVI quantizes keys per channel and values per token, and keeps a full-precision recent window. It is the most direct competitor: it is exactly the configuration our correction plugs into, and it reports near-lossless clean results. The file cites CoQA for Llama-2-13B at 66.37 in 16-bit versus 66.23 with KIVI-2bit, and about a 2% drop on GSM8K. No reported result was found on Lacuna for KIVI on RULER or under distractor load. We run it with its released group size and window settings on identical prompts, and add our correction as the only change.
KIVI plus KVSink sink protection. This is the strongest structured-quantization baseline. It is also the configuration where our hypothesis predicts the worst sink drain, so it tests the claim directly.
AnchorKV. AnchorKV is the strongest reported method on the file's long-context benchmark. It retains 99.3% of the uncompressed RULER score on Llama-3.1-70B at 20× compression, where the strongest other baseline retains 86.8%. We compare at matched effective bits, including its full-precision anchors and index metadata, on the 8B model and on the 70B check.
Metadata-free quantizers. NSNQuant and QJL are run at their reported effective bit-widths, about 1.23 bits for 1-bit NSNQuant and 3 bits for QJL. They test whether removing scales is better than keeping cheap scales and correcting.
Uniform offset ablation. A tuned, query-independent constant is subtracted from all quantized-token logits. It isolates whether our correction's dependence on the query and on each group's step sizes matters. The constant gets the same tuning budget, a grid search on a held-out split, as any hyperparameter of ours. Our correction itself has no tuned parameters.
Beating a baseline means a statistically significant reduction in the amplification gap at $N \ge 64$ at equal or lower effective bits, with clean-prompt accuracy no worse than the baseline's.
Why not just…?
Why not just use finer groups, or stay at 4 bits? This is the real competitor. More metadata reduces both the bias and the variance. The question is whether it does so at a lower total bit cost than correction. Stage 4's matched-bits frontier settles this directly. If uncorrected fine groups dominate the frontier, the budget claim fails and we report the diagnosis alone. Staying at 4 bits caps compression at roughly 4–5× per AnchorKV's analysis. Yazan et al. also found LLaMA2-7B degrading sharply with context under 4-bit weights, so 4 bits alone does not guarantee robustness.
Why not just filter or rerank the retrieved context first? Rerankers and structured prompting, like Superposition Prompting, which reports 43% accuracy gains on NaturalQuestions, reduce $N$ and would shrink the amplification we describe. They are complementary, not a replacement. Many long-context tasks, such as multi-hop tracing in RULER or coherent distractors inside a single passage in Breaking Focus, cannot be cleanly filtered, and filtering does nothing for cache memory. We add a condition with a reranker in front, to measure how much gap remains after filtering.
Why not just evict the distractors from the cache? Eviction decides relevance before the question is known. AnchorKV reports eviction collapsing to a 0.18 retrieval score on 128K Needle-in-a-Haystack versus 0.94 for its method, and The Pitfalls of KV Cache Compression documents systematic loss of instructions under eviction. We include SnapKV-style eviction at matched memory as a reference, but we expect it to fail on exactly our harness.
Why not just quantize the sinks too, so nothing is favored? This would remove the imbalance, but KVSink shows that quantizing sinks collapses perplexity. We run it anyway as a one-line ablation, because its failure pattern also helps confirm that sink mass is the currency being transferred.
Decisive Checks
The hypothesis is falsified if either of the following holds: - the amplification gap does not grow with $N$ at 2 bits; - the predicted mass transfer does not correlate with measured sink-mass loss across heads.
The correction claim fails if either of the following holds: - the uniform offset ablation matches it; - corrected 128-token groups do not match uncorrected 32-token groups on RULER QA at $N=256$. The corrected configuration would sit roughly 0.4 bits per value lower, from a 16-bit scale shared over 128 versus 32 tokens.
Known limits. Two failure cases are anticipated. The stealing paper reports that the correction breaks when attention is sharply peaked, so we report results separately by head sharpness. And decode-time behavior is exactly where that paper expected weak transfer, so we separate prefill from decode. The most informative negative outcome would be that the random, extreme-value spread dominates the gap. That would mean distractor robustness genuinely requires metadata, and that zero-overhead schemes like QJL are at particular risk on noisy prompts.
References
- Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- RULER: What’s the Real Context Size of Your Long-Context Language Models?
- The Impact of Quantization on Retrieval-Augmented Generation: An Analysis of Small LLMs
- NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
- QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead
- LO-BCQ: Locally Optimal Block Clustered Quantization for 4-bit (W4A4) LLM Inference
- BiE: Bi-Exponent Block Floating-Point for Large Language Models Quantization
- AnchorKV: Anchor-Residual KV Cache Compression
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- The Pitfalls of KV Cache Compression