TL;DR
While scaling up large language models (LLMs) typically improves performance across most tasks, this paper uncovers a striking counter-example: the Size-Fidelity Paradox in context compression. When compressing long text into compact memory tokens, larger models actually do a worse job of faithfully preserving exact source details than mid-sized models, frequently overwriting rare facts with their own internal priors or distorting fine-grained semantic relationships.
What Problem Does This Paper Address?
Context compression is an increasingly vital technique used to pack long documents into a small set of continuous latent embeddings (memory tokens), accelerating decoding and saving memory. Standard scaling laws dictate that larger models make better compressors, a notion supported by the fact that larger models achieve lower training loss and higher surface-level text similarity scores (like BLEU or ROUGE).
However, the authors demonstrate that these standard metrics are deceptive. They reward fluency and surface similarity but fail to capture when a model subtly alters critical details. This paper addresses the breakdown of scaling laws when the objective shifts from generating plausible text to faithfully preserving exact source information.
Methods
The authors evaluate context compression within a standard compressor-decoder paradigm: a compressor LLM maps an input sequence into continuous memory tokens, and a decoder LLM attempts an exact reconstruction of the original text.
To expose hidden fidelity failures that standard metrics miss, the authors design a novel usage-oriented evaluation framework featuring two diagnostic tasks:
- Knowledge Overwriting Probe: Evaluates whether a compressor prioritizes the source text over its own internal parametric knowledge by feeding it counterfactual contexts (e.g., "Einstein was born in France") and testing its factual recall.
- Semantic Drift Probe: Evaluates structural changes during compression (such as role reversals or modifier shifts) by using an advanced model (DeepSeek-R1) to generate fine-grained relational questions requiring exact substring answers from the source text.
Additionally, they probe the hidden memory embeddings using effective rank to analyze embedding geometry and conditional entropy to measure reconstruction determinacy under teacher forcing.
Datasets
- Training & Base Reconstruction: High-quality text chunks from the FineWeb dataset.
- Knowledge Overwriting Evaluation: Controlled counterfactual variants of FaithEval and ConflictQA.
- Semantic Drift Evaluation: A newly generated diagnostic QA dataset built over FineWeb and FaithEval.
- Downstream Tasks: ArxivCorpus (for cross-domain robustness), MuSiQue and QASPER (for retrieval-augmented generation), and subsets of Big-Bench Hard (BBH) (for in-context learning).
Key Insights and Ideas
- The Size-Fidelity Paradox: As a compressor model scales up, its surface-level reconstruction scores remain high, but its ability to preserve source-specific facts and relations degrades. Mid-sized models (around the 3B–4B parameter range) consistently emerge as the most stable and faithful compressors.
- Knowledge Overwriting: Larger models possess incredibly strong internal world priors. When forced to compress a rare or counterfactual fact, the model's internal beliefs override the source text during reconstruction (e.g., changing "the white strawberry" to "the red strawberry").
- Semantic Drift: Larger models excel at abstract semantic compression—capturing the general "gist" of a sentence—but they lose sensitivity to fine-grained relational constraints. This causes them to paraphrase text in ways that flip causal roles (e.g., changing "the bee vibrates its muscles to shake pollen from flowers" into "the flower vibrates its anthers to shake pollen onto bees").
- The Underlying Mechanism: Embedding analysis reveals that larger compressors diffuse memory across broader semantic subspaces (higher effective rank) and yield higher conditional entropy. This means their compressed representations are more ambiguous, presenting multiple plausible alternative paths to the decoder rather than strictly anchoring the exact source text.
Key Results
- Non-Monotonic Performance: Across 27 compressor setups spanning the Qwen3 and LLaMA 3.2 families (from 0.6B to 90B parameters), accuracy on the diagnostic QA tasks peaked at mid-sized scales and fell sharply for the largest models.
- Fidelity vs. Surface Metrics: At 64× compression, LLaMA-3.2 90B achieved a higher BLEU score than LLaMA-3B (0.33 vs. 0.29) but suffered a severe drop in knowledge overwriting accuracy (0.53 vs. 0.62).
- Downstream Impact: In practical applications like Retrieval-Augmented Generation (RAG) on MuSiQue and QASPER, the 8B and 32B Qwen3 compressors significantly underperformed the smaller 0.6B and 1.7B variants because they failed to preserve source-specific multi-hop evidence.
Limitations and Future Work
- Architecture Bounds: The conclusions are tied to continuous memory-token compressor-decoder setups. Other paradigms, such as extractive compression or architectures with explicit copy mechanisms, might scale differently.
- Diagnostic vs. Causal Probes: The effective rank and conditional entropy analyses reveal strong, consistent associations with fidelity failures, but they do not provide an immediate algorithmic intervention to fix large models.
- Future Directions: The authors highlight the need to develop new training objectives for large language models that explicitly strengthen source anchoring and enforce rigid fidelity, ensuring that "more" parameters can truly deliver "more" accuracy.
Summary updated