TL;DR
This paper introduces a framework for "translating" the internal conversation memory (KV cache) between different large language models, even when they use radically different architectures like compressed latents (MLA) or recurrent states (Linear Attention). By mapping these caches directly, the authors achieve up to a 46.8x speedup in model hand-offs compared to re-reading the prompt, while maintaining the target model's native output quality.
What Problem Does This Paper Address?
In modern AI workflows, we often switch models mid-conversation—perhaps starting with a small, fast model for simple turns and escalating to a massive "flagship" model for complex reasoning. Normally, the second model must re-read the entire conversation history (the "prefill" stage) to build its internal state. This is slow, expensive, and scales poorly with long contexts.
While previous research like KV-Lingo enabled cache translation for standard models, the "frontier" has moved on. New models like DeepSeek-V3 and Qwen3.5 use complex hybrid architectures:
- Multi-head Latent Attention (MLA): Caches compressed vectors instead of raw keys/values.
- Linear Attention / SSMs: Replace per-token caches with fixed-size recurrent states.
- Sparse Attention: Use "indexers" to decide which tokens to even look at.
These innovations break traditional translation methods. This paper provides the "Rosetta Stone" for these modern, hybrid cache objects.
Methods
The authors propose a two-stage translation pipeline:
- Stage 1 (Linear Alignment): They fit per-layer linear maps in closed form using a few hundred thousand tokens of paired data.
- Stage 2 (KL Distillation): They fine-tune the translator to minimize the Kullback–Leibler (KL) divergence between the target model's output using a translated cache versus its own native cache.
Per-Object Recipes
The core contribution is a set of specific strategies for different cache types:
- MLA Latents: Mapped token-by-token using linear layers. Crucially, they capture the data before positional rotation (RoPE) to keep the map position-independent.
- Recurrent States: If the source and target have the same state shape, they use a "copy + bilinear correction" method. If the source lacks states entirely, they "replay" the target's own recurrence over predicted per-token inputs.
- Token Identity: The authors discovered that a linear map alone isn't enough for perfect long-range recall. They decode token IDs directly from the source's first MLA layer (which acts as a deterministic signature of the token) and feed these IDs back into the translation maps.
Datasets
- Training/Validation: The models were trained and validated on chat conversations from the
Nemotron-SFT-Instruction-Following-Chat-v2dataset. - Evaluation Probes:
- CoQA: Conversational Question Answering.
- ARC-Challenge: Hard-set reasoning questions.
- Needle-in-a-Haystack: Testing the model's ability to recall a specific fact buried in up to 128K tokens of context.
- MT-Bench: Using a "judge" model (Gemma-2-31B) to rate chat quality.
Key Insights and Ideas
- The MLA "Inversion" Trick: One of the most elegant findings is that the first layer of an MLA model effectively stores the token's identity. By "reading back" the token IDs from the cache, the translator can provide the target model with an exact signal, boosting "needle" recall from a dismal 0.09 to a near-perfect 0.99.
- Capture Points are Critical: You can't just translate the final tensor the engine stores. To achieve high fidelity, you must capture the "raw" projection outputs before they are normalized or rotated by positional embeddings.
- Segment-wise Multi-turn: Naively re-translating the whole history at every turn causes quality to collapse. Instead, translating only the new segment and appending it to the existing translated cache maintains "native" performance over long dialogues.
Key Results
The researchers tested their methods on several model pairs, including Kimi-VL-A3B ↔ Kimi-Linear-48B and Qwen3.5-27B ↔ Qwen3.8-27B.
- Speed: At 64K tokens, the "switch" time was 18.7x to 46.8x faster than a standard re-read. For a Qwen pair, the time to first token dropped from 6.8 seconds to just 208 milliseconds.
- Quality: In the "Hybrid → Full-MLA" direction (e.g., handing off from a fast hybrid to a flagship), the translated model was statistically indistinguishable from the native model on every task metric.
- Efficiency: MLA latents are significantly smaller than standard GQA caches, making the transfer even more efficient.
Limitations and Future Work
- Scale: While tested on models up to 48B parameters, the authors state that feasibility for 300B+ "flagship" models remains a hypothesis (though a strongly supported one).
- Tokenizer Dependency: Currently, the source and target models must share the same tokenizer. Cross-family transfer (e.g., DeepSeek to Kimi) would require an additional "token aligner" layer.
- Behavioral Leakage: The authors noted that translated states can sometimes carry over the "personality" or safety refusals of the source model—a phenomenon they call behavioral leakage.
- Sparse Attention: The work on sparse-attention indexers was a "rehearsal" at a smaller scale and needs further validation on massive production models like DeepSeek-V3.
Summary updated