One Compacted Cache, Many Samples: Mass-Preserving KV Compaction for Long-Document Data Synthesis

Long-context training data is usually made by having a large "teacher" model read an entire book and write many questions, claims, or reasoning traces about it, which a smaller "student" model is then trained on. The expensive part is not reading the book once. It is the hundreds of generations that each have to attend over the book's full key-value cache. That cache is the stored internal representation of every token the model has read, and at book length it fills GPU memory and caps how many generations can run side by side. This proposal compresses that cache once per book, using proportional attention weighting. When many stored tokens are replaced by a few representatives, each representative gets an additive $\log s$ bias in the attention scores, so it keeps the attention weight of the $s$ tokens it stands for. Hundreds of generations then decode against the small cache at batch sizes the full cache cannot support.

Synthesis also helps the compression. The best fast compaction method, Fast KV Compaction via Attention Matching, fits its compressed cache to a set of "reference queries" that probe how the model will attend. It names the cost of producing realistic reference queries as a main open bottleneck. In synthesis, the teacher's first few generations from the full cache give exactly these queries, and they are not wasted, because those generations are kept as data. The research question is whether data made this way trains students as well as data from today's text-summarization pipelines, at lower teacher cost, without quietly skewing which parts of the book the data covers.

Core Claim

For corpora with many samples per long document, mass-preserving cache compaction is the right place to cut teacher cost. The compaction is fitted on the teacher's own pilot generations and shared across all later samples from the same document. The proposal makes three predictions:

  1. It should beat full-cache generation with prefix sharing on throughput per GPU-hour.
  2. It should beat uncompensated token eviction on data quality, and especially on coverage of the whole document.
  3. Combined with a short text outline, it should match or exceed text-space compression pipelines on downstream student accuracy, because it keeps fine-grained detail that outlines discard.

The closest prior work evaluates compaction only for single-query inference, not as a shared substrate for large-scale data generation. Its fidelity has never been measured by what a student learns from the resulting data.

Where the Pieces Come From

Proportional weighting methods supply the mechanism. - Token Merging: Your ViT But Faster merges similar image tokens and adds a size term to the softmax. It doubles vision-transformer throughput at a 0.2–0.3% accuracy cost. - Token Merging for Fast Stable Diffusion shows that merging speeds up a generator, not just a classifier. It reports FID 33.02 versus 33.12 at 50% token reduction. It stays within image diffusion and never touches text generation or distillation. - Homogeneous Keys, Heterogeneous Values, called AsymKV, merges correlated adjacent keys and sums values. It keeps a count vector so the softmax denominator stays correct. - Attention Matching selects keys with Orthogonal Matching Pursuit and fits per-key biases with non-negative least squares. It refits values with ordinary least squares and reaches 50× compression in seconds.

The proposal reuses Attention Matching as its compaction step, with AsymKV as a second variant. It inherits two documented limits from these papers: quality collapses beyond roughly 100× compression, and the per-head formulation does not fit multi-head latent attention. The proposal therefore stays below 100× and uses grouped-query-attention teachers.

Text-space synthesis pipelines motivate the work and are the incumbent. - CLIPPER first compresses books into chapter outlines and summaries, then generates claims from them. - Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation chunks books and summarizes them hierarchically with Qwen-2-72B before writing 580,000 question-answer pairs.

Both pay extra generation passes to build their summaries, and both lose token-level detail. Neither touches the teacher's internal attention.

One result constrains the design. CLIPPER reports that naive generation from uncompressed books gives a 73.1% claim error rate, against 16.7% with outlines. A compacted cache that faithfully imitates full-book attention should inherit the full-book behavior, so it could also inherit the high error rate. Pure latent compaction is therefore not assumed to replace outlines. It is tested both alone and as a detail channel next to an outline.

A safety result motivates a coverage audit. The Pitfalls of KV Cache Compression shows that eviction methods such as SnapKV and StreamingLLM selectively forget particular instruction spans. The data-generation version of that failure is a corpus that silently over-represents some chapters. Teacher-fidelity scores would not catch it, so the proposal measures it directly.

Research Program

Step 1: prefill and pilot. For each book, the teacher (an open grouped-query-attention model such as Qwen-2-72B) prefills the full text once. It then produces a small pilot batch of $K_0$ generations, for example 16 claims or questions, using the full cache. The query vectors produced while writing these pilots are logged at every layer and head.

Step 2: fit the compacted cache. Call the logged pilot queries $\{q\}$. Attention Matching picks a subset $S$ of the book's keys. It then fits a bias $\beta_j \ge 0$ for each kept key $j$ so that the compacted cache reproduces the full cache's total attention mass:

$$\min_{\beta \ge 0} \sum_{q} \Big( \sum_{m=1}^{n} e^{q \cdot k_m/\sqrt{d}} - \sum_{j \in S} e^{q \cdot k_j/\sqrt{d}} \, \beta_j \Big)^2 .$$

Here $n$ is the number of book tokens, $k_m$ is the key of token $m$, and $d$ is the head dimension. In the softmax, $\beta_j$ acts as $e^{\log s_j}$, an effective count of the tokens that key $j$ stands for. Values are then refit by least squares.

Step 3: generate at scale. All remaining $K$ generations, for example 500 per book, decode against the compacted cache at large batch size. The fitting cost and the pilots are paid once and spread over all $K$ samples.

What is varied. The experiments vary:

  • the compression ratio (10×, 25×, 50×, 100×);
  • the source of reference queries (on-policy pilots, generic prompt queries, or random queries);
  • whether the $\beta$ bias is used;
  • whether a CLIPPER-style outline is placed in the prompt as text ahead of the compacted book cache.

What is measured. - Teacher GPU-hours per accepted sample, with every cost counted: prefill, pilots, fitting, decoding, and filtering. - Claim error rate, using the same validator protocol in every arm. - Evidence coverage: the distribution of each generated sample's supporting passage over book position. - Downstream student accuracy.

Baselines

Naive reference point. The teacher generates every sample directly from the full book with no compression and no outline. This is CLIPPER's uncompressed condition, which it reports at a 73.1% claim error rate. It bounds what faithful full-context imitation can achieve on quality.

Full-cache prefix sharing (strongest efficiency competitor). The full book cache is prefilled once and shared across all generations in a serving system. Batch size is pushed as high as memory allows. This baseline already removes repeated prefill, so it isolates the remaining benefit of compaction: higher batch size and cheaper per-token attention during decoding. It runs on identical hardware with the same prompts and sample counts, and its batch size is tuned to the memory limit. Compaction beats it only if it delivers at least 3× more accepted samples per GPU-hour with error rate and coverage within noise. The file gives no reported throughput number for this setting.

Uncompensated eviction at matched cache size. SnapKV-style and H2O-style eviction are run at the same retained-token budget, with no $\beta$ fitting. AsymKV reports a LongBench average of 43.95 against 38.89 for H2O on Llama-3.1-8B, and 39.39 against 14.75 on synthetic needle retrieval. These numbers say eviction should lose, but they come from single-query inference, not corpus generation.

Ablations of the key ingredient. - The same compaction with all $\beta$ set to zero isolates the proportional weighting. - The same compaction fitted on generic or random queries isolates the on-policy pilot idea.

CLIPPER (strongest quality competitor). CLIPPER reports a 16.7% claim error rate and $0.05 per claim. A Llama-3.1-8B-Instruct student trained on its claims reaches 76.0% on the CLIPPER test set, up from 27.9%. For Attention Matching itself on RULER, no reported result was found on Lacuna. The fair comparison runs CLIPPER's outline step with the same teacher and counts the outline-generation compute. It trains the same student on an equal number of accepted claims with an identical fine-tuning budget. The goal is to match 76.0% within two points at lower total teacher compute, or to exceed it at equal compute.

Why not just…?

Why not just use full-cache prefix sharing, which is lossless? This is the most serious threat. For short documents, such as the passages Promptagator turns into queries in Promptagator, it probably wins outright. That is why the proposal is restricted to book-length inputs, where cache memory rather than compute limits batch size. Experiment 1 measures the crossover document length directly. If compaction gives no gain above 100K tokens, the efficiency claim fails.

Why not just use eviction such as SnapKV, which needs no fitting? Fitting with least squares takes seconds per document, and that cost is spread over hundreds of samples, so eviction's cost advantage is negligible here. Eviction's documented failures matter more. The Pitfalls paper shows it selectively forgets spans, and AsymKV shows it scores near zero on early-context retrieval. In synthesis this would show up as a corpus that ignores early chapters. The coverage audit in Experiment 2 tests this.

Why not just keep CLIPPER's text outlines, which are cheap and reduce hallucination? For error rate, outlines may truly be the better tool. Attention-space compaction alone is not expected to beat 16.7%, for the reason given in the CLIPPER constraint above. Outlines, however, cap claims at whatever the outline mentions. The outline-plus-compacted-cache arm tests whether adding latent detail increases claim specificity and student accuracy on NoCha and NarrativeQA without raising error. If it does not, the result is reported as a negative finding about latent compression for data quality.

Why not just use a smaller teacher? TrueTeacher needed a 540B teacher to produce labels from which an 11B student reached 87.8 ROC-AUC. This suggests teacher capacity is often not negotiable. Compaction keeps the large teacher and shrinks only its memory footprint. Even so, an 8B teacher with full cache at equal GPU-hours is included as a control.

Decisive Experiments

Experiment 1: efficiency frontier. Generate claims from 200 books spanning 50K–300K tokens. Plot accepted samples per GPU-hour against claim error rate for:

  • full-cache prefix sharing;
  • eviction;
  • compaction with and without $\beta$;
  • compaction with each reference-query source.

Compaction should dominate eviction at every ratio and should overtake full-cache sharing above some document length, which this experiment measures.

Experiment 2: coverage audit. Compute the KL divergence between the book-position histogram of supporting evidence under compressed caches and the histogram under the full cache. The prediction is that $\beta$ fitting keeps this divergence near the full-cache reference while eviction shows a measurable shift. This is the data-level analogue of selective amnesia.

Experiment 3: student outcome. Fine-tune Llama-3.1-8B-Instruct on equal-size corpora from four arms:

  • CLIPPER;
  • compacted cache alone;
  • outline plus compacted cache;
  • full-cache sharing.

Evaluate on the CLIPPER test set, NoCha, NarrativeQA, and InfiniteBench QA. For context on InfiniteBench, the hierarchical pipeline reports 59.45 from 180K synthetic samples against a 51.31 base. Also check MMLU for regressions; the hierarchical pipeline saw a drop from 68.21 to 65.08.

What would sink the idea. Any of three outcomes would sink it:

  • the outline-plus-cache arm fails to match CLIPPER's student at lower total compute;
  • on-policy pilots give no gain over generic reference queries;
  • the advantage disappears once full-cache batching is tuned properly.