Long-context language models store a key and a value vector for every past token. This "KV cache" grows with the conversation and quickly dominates GPU memory. Two families of methods shrink it. The first drops tokens: after reading the prompt, a heuristic decides which tokens the current question pays attention to and throws the rest away. The second compresses: the model is trained to fold each short chunk of tokens into a single summary that it can still attend to later. A recent benchmark that reuses one long document across many follow-up questions, SCBench, found that token-dropping methods with less-than-linear memory work on the first question and then collapse on later ones. The tokens they discarded were exactly what later questions needed. SCBench calls this the "sub-$O(n)$ memory trap." It never tested learned chunk compression.
This note proposes one controlled experiment that separates two explanations of the trap. The first explanation is that any method keeping much less than one entry per token must fail when the questions change. The second is that the failure comes specifically from choosing what to keep based on the current question, before the future questions are known. Chunk-summary models such as RAT compress without looking at any question, so they are a natural test case. We hold memory per token fixed and pretrain every architecture under the same recipe. Then we measure how accuracy decays across turns. If learned, question-blind compression stays flat while question-driven eviction at the same memory budget decays, then the trap is a selection-timing problem, not a capacity problem. That answer would change how compressed long-context models should be designed.
Core Claim
At a fixed KV memory budget of roughly one sixteenth of full attention, consider a decoder that writes one learned summary per 16-token chunk while reading the prompt. We claim it keeps its first-turn accuracy across many follow-up questions on a shared document. Attention-guided eviction applied to a full-attention model at the same budget does not. We further claim that most of the eviction gap closes if the eviction heuristic is allowed to see all future questions in advance. This oracle variant is the ablation that isolates the key ingredient. If RAT decays as fast as eviction, the capacity explanation wins and the claim is falsified.
The claim deliberately goes against one result in the file. KV Cache Compression, But What Must We Give in Return? reports that compressing during prompt ingestion ("prefill") compounds errors across layers, while compressing only during decoding is safer. RAT compresses entirely at prefill. That paper only studied post-hoc compression of models that were never trained to compress. Our study tests whether this "sanctity of prefill" still holds once compression is learned during pretraining. We treat it as a competing prediction, not as settled.
Relation to Prior Work
Three papers supply the motivation. SCBench introduced the multi-turn and multi-request protocols and found the trap across 13 post-hoc methods on eight models. However, it never evaluated chunk-recurrent architectures, and the file contains no specific numeric task scores from it. The Pitfalls of KV Cache Compression showed that global eviction preferentially discards system-prompt instructions, which leaks hidden prompts. That is another symptom of question-driven selection, and our design can test it at no extra cost. RULER showed that passing a single needle-retrieval test says little about variable tracking or aggregation, so we do not rely on simple needle tests.
Two papers supply the methodological template. Simple Local Attentions Remain Competitive for Long-Context Tasks pretrained many efficient-attention encoders under one recipe. It found that rankings on the synthetic Long Range Arena (about 60–61% for everyone) hid large downstream gaps, for example 72.9 versus 31.9 TriviaQA F1 for local windows versus Performer. Revisiting Point Cloud Shape Classification showed that fixing the training protocol alone raised PointNet++ from 90.0% to 93.3% on ModelNet40. Both warn that uncontrolled recipes manufacture architectural wins. The local-attention study was encoder-only and capped at 4,096 tokens. Ours is decoder-only, evaluated on multi-turn reuse, and matched on memory rather than on FLOPs.
The methods under test come from three papers. RAT uses within-chunk linear recurrence with attention across chunk summaries, at chunk length 16. It reports 7× training speedup at 100K length and 9× generation speedup at 4K, at 1.3B scale. It was evaluated only on single-turn tasks against baselines it tuned itself. ChunkKV evicts 10-token chunks by attention score and reports up to 8.7% gains over SnapKV and H2O at a 10% compression ratio. It is still question-driven, which makes it the clean middle point between token eviction and learned summaries. Mesa-Extrapolation is an inference-time position-encoding method, not a memory reducer, so we leave it out.
Research Program
We first pretrain a family of roughly 1.3B-parameter decoders on the same FineWeb-Edu token stream (FineWeb), following RAT's setup but at a reduced budget of around 50B tokens per arm. Every arm uses the same tokenizer, learning-rate schedule, batch size and data order. Every arm then gets the same long-context continuation to 32K tokens. The arms are full attention, sliding-window attention, RAT with chunk length 16, and the RAT-with-sliding-window hybrid that RAT reports as its best. The last arm is a RAT variant whose learned chunk summary is replaced by average pooling of the chunk's keys and values, in the spirit of the pooling result of Burchi and Timofte. This variant tells us whether a learned summary matters or whether any question-blind reduction suffices.
Each arm gets the same hyperparameter search budget, selected on a held-out validation split and never on test data. On the trained full-attention model we then apply the post-hoc methods at matched memory: SnapKV and H2O-style token eviction, StreamingLLM, ChunkKV, and 4-bit KV quantization (KIVI).
The central measurement is a retention curve. It plots accuracy at turn $k$ divided by accuracy at turn 1, against memory per token measured in bytes. We record it on SCBench's multi-turn and multi-request modes across its four task categories. Because our models reach 32K rather than 128K tokens, we regenerate SCBench tasks at 32K. We will report this deviation and release the generation scripts. Secondary evaluations are RULER at 4K–32K, LongBench, the 64-digit passkey stress test from KV Cache Compression, But What Must We Give in Return?, and the system-prompt leakage probe from The Pitfalls of KV Cache Compression. All latency and memory numbers are wall-clock and peak VRAM on the same GPU and serving stack, not FLOP counts.
Baselines
The naive reference point is full attention with an uncompressed cache. It defines the accuracy ceiling and the memory cost that every other arm is trying to undercut. Per SCBench's finding about linear-memory methods, it should give a flat retention curve.
The strongest training-free competitor is 4-bit KV quantization (KIVI) applied to our full-attention model. KV Cache Compression, But What Must We Give in Return? reports 45.3% on LongBench against 45.2% uncompressed, and 100% needle retrieval on Llama-3, which makes it the most robust method in the file. It keeps every token, so it should not fall into the multi-turn trap. However, it reaches only a 4× reduction. We therefore compare it at its native budget, and we also report the RAT variant with chunk length 4 so the two meet at matched bytes per token. Beating KIVI means matching its retention curve at 4× while extending to 16× where KIVI cannot go.
The strongest eviction competitors are SnapKV and ChunkKV at a 1/16 budget on the same full-attention checkpoint. ChunkKV's reported gain of up to 8.7% over SnapKV at 10% compression makes it the best question-driven method in the file. Beating them means a turn-5 retention ratio for RAT that is clearly higher, with non-overlapping seed intervals across three evaluation seeds, at equal or lower peak memory.
Sliding-window attention with a window sized to the same memory is the simplicity baseline that Simple Local Attentions Remain Competitive for Long-Context Tasks says we must take seriously. It is question-blind but discards distant context entirely, so it separates question-blindness from long-range access.
The decisive ablation is oracle SnapKV, which scores tokens using the concatenation of all future turns' questions. If oracle SnapKV recovers RAT-level retention, the trap is about when the selection is made. If it does not, then question-blind learned summaries are doing something that clever selection cannot.
Why not just…?
Why not just run the released RAT checkpoint through SCBench? This is cheap and we will do it first as a pilot. It cannot settle the question, because any difference from Llama-class eviction baselines is confounded with data, scale and recipe. The point-cloud and local-attention studies both show that this confound can reverse a ranking. A pilot showing that RAT collapses would still be informative and would kill the project early.
Why not just use 4-bit quantization and accept 4×? This is a real threat. If practitioners only need 4×, KIVI's reported parity on LongBench means a new architecture is not justified. Our claim is restricted to budgets of 8× and beyond, where quantization alone cannot reach. The chunk-length-4 RAT arm checks whether chunking costs anything at the regime where KIVI already works. If RAT loses there, we will say so.
Why not just give eviction a larger budget, or keep sparse computation with full memory as MInference does? SCBench shows that linear-memory sparse methods are stable, but they do not save memory, which is the goal here. Larger eviction budgets trace out a curve that we will plot in full. The question is whether eviction's curve lies below RAT's at every budget, not whether eviction can be rescued by spending memory.
Why not just use sliding windows plus a few global tokens? In the encoder setting, local windows with a retained global token matched overlapping windows at more than 2× throughput (Simple Local Attentions Remain Competitive for Long-Context Tasks). We include the sliding-window arm, and a variant with sink tokens via StreamingLLM. RULER-style variable tracking at 32K should expose whether a few global slots are enough. If they are, the simpler design wins.
Decisive Checks
The experiment succeeds if three things hold at 1/16 memory. RAT's SCBench retention ratio at turn 5 stays near full attention's. SnapKV and ChunkKV fall clearly below it. Oracle SnapKV closes most of that gap. The capacity explanation wins if RAT decays like eviction. The prefill-sanctity prediction wins if RAT's turn-1 accuracy already trails eviction, meaning compression at prefill hurts even when learned. The pooled-summary arm shows whether learning the summary matters. The leakage probe tests a corollary: question-blind compression should not preferentially lose system instructions. No reported result was found on Lacuna for RAT or ChunkKV under SCBench, or for SCBench numeric task scores, so all of these comparisons are new measurements.
References
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling
- KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
- The Pitfalls of KV Cache Compression
- RULER: What’s the Real Context Size of Your Long-Context Language Models?
- Simple Local Attentions Remain Competitive for Long-Context Tasks
- Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
- Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs
- Audio-Visual Efficient Conformer for Robust Speech Recognition