TL;DR
TraceKV is a training-free framework designed to compress the massive Key-Value (KV) caches required by reasoning-heavy Vision-Language Models (VLMs). By shifting the focus from simple data reconstruction to "policy preservation"—using on-policy calibration and sensitivity-aware rank allocation—TraceKV achieves up to 5.64× compression with near-lossless accuracy and delivers a 1.83× decoding speedup.
What Problem Does This Paper Address?
As Vision-Language Models (VLMs) move toward complex multi-step reasoning (like visual math or scientific analysis), they generate lengthy "Chain-of-Thought" traces. This creates a massive memory bottleneck: the KV cache must store both the high-resolution visual inputs and the ever-growing reasoning text.
Existing low-rank compression methods (which factorize KV matrices to store smaller "latents") work well for short-form tasks but fail during extended reasoning. The authors identify two primary culprits for this failure:
- Calibration-State Mismatch: Standard methods calibrate using static input data. However, the model's self-generated reasoning tokens follow a different distribution, leading to high reconstruction errors as the reasoning chain grows.
- Unequal Error Sensitivity: Not all tokens are created equal. A small error in a redundant visual token might not matter, but the same error in a critical logical step can cause the model to hallucinate, loop infinitely, or fail entirely.
Methods
TraceKV introduces a unified policy-preserving principle to minimize the divergence between the original "dense" model and the compressed version. Its core components include:
- On-Policy Calibration: Instead of using fixed datasets, the authors let the dense VLM generate its own reasoning traces for the calibration set. This ensures the compression algorithm "sees" the actual states the model encounters during inference.
- Sensitivity-Aware SVD: The team uses Fisher Information to quantify how much a perturbation in a specific KV state affects the model's final output policy (the next-token probability distribution). They then weight the SVD objective so that highly sensitive tokens (like logical connectors or "sink" tokens) are reconstructed with higher fidelity.
- Global Rank Allocation: Rather than giving every layer the same compression ratio, TraceKV dynamically allocates the "rank budget" across different layers and projections (Key vs. Value) based on their importance to the overall reasoning policy.
- System Optimization: To make this practical, they developed custom Triton kernels that perform on-the-fly KV reconstruction and RoPE application, avoiding the need to ever materialize the full, uncompressed cache in GPU memory.
Datasets
The authors evaluated TraceKV across several challenging multimodal reasoning benchmarks:
- ScienceQA: Scientific reasoning in a visual context.
- MathVista: Mathematical reasoning involving charts, diagrams, and images.
- MMMU: A massive multi-discipline benchmark for expert-level artificial general intelligence.
Calibration was performed using a small subset (256 samples) of the ScienceQA training split.
Key Insights and Ideas
- Reconstruction $\neq$ Reasoning: Minimizing Mean Squared Error (MSE) in the KV states doesn't guarantee the model will still "think" correctly. Preserving the Policy-KL divergence is a much better proxy for maintaining reasoning performance.
- Visual Redundancy: The analysis revealed that many visual tokens have very low sensitivity. TraceKV exploits this by allocating less memory to these regions while protecting critical textual reasoning tokens.
- Layer Heterogeneity: Importance isn't uniform across the model. The authors found that Value projections generally require more rank than Key projections, and middle-to-late layers (specifically layers 16–21 in a 7B model) are more sensitive to compression errors.
Key Results
- Near-Lossless Compression: On Qwen2.5-VL-7B, TraceKV achieved 73.33% average accuracy at a 70% rank ratio, nearly matching the dense model's 73.58%.
- Synergy with Quantization: When combined with 4-bit quantization, TraceKV reaches 5.64× compression with negligible accuracy loss. At 50% rank and 4-bit quantization (7.76× compression), it still outperforms all other low-rank baselines by a wide margin.
- Efficiency Gains: At a 16K context length, the optimized kernels provide a 1.53× attention speedup and a 1.38× end-to-end speedup, while reducing peak GPU memory usage by nearly half (from ~32GB to ~18GB).
- Scalability: The method was successfully tested on larger models like Qwen3-VL-32B-Thinking, where it maintained high accuracy even under aggressive compression.
Limitations and Future Work
- Distribution Shift: TraceKV minimizes KL divergence relative to the dense model's traces. However, during real-world inference, the compressed model might drift onto its own unique traces. The authors suggest exploring objectives that account for the compressed model's own autoregressive states.
- Calibration Overhead: Generating on-policy traces and computing Fisher information takes time (roughly 30 minutes for a 7B model on a single L40S GPU). While this is a one-time offline cost, reducing this overhead remains a goal.
- Generalization: While the method generalizes well across the tested reasoning benchmarks, its performance on extremely long-form creative generation or non-reasoning tasks hasn't been fully explored.
Summary updated