Translating KV Caches Between Hybrid-Attention Language Models: MLA Latents, Linear-Attention States and Sparse-Attention Indexers

Povilas Kvedaras, Tomas RasymasarXiv 2026 · arXiv

TL;DR

This paper introduces a framework for "translating" the internal conversation memory (KV cache) between different large language models, even when they use radically different architectures like compressed latents (MLA) or recurrent states (Linear Attention). By mapping these caches directly, the authors achieve up to a 46.8x speedup in model hand-offs compared to re-reading the prompt, while maintaining the target model's native output quality.

What Problem Does This Paper Address?

In modern AI workflows, we often switch models mid-conversation—perhaps starting with a small, fast model for simple turns and escalating to a massive "flagship" model for complex reasoning. Normally, the second model must re-read the entire conversation history (the "prefill" stage) to build its internal state. This is slow, expensive, and scales poorly with long contexts.

While previous research like KV-Lingo enabled cache translation for standard models, the "frontier" has moved on. New models like DeepSeek-V3 and Qwen3.5 use complex hybrid architectures:

  • Multi-head Latent Attention (MLA): Caches compressed vectors instead of raw keys/values.
  • Linear Attention / SSMs: Replace per-token caches with fixed-size recurrent states.
  • Sparse Attention: Use "indexers" to decide which tokens to even look at.

These innovations break traditional translation methods. This paper provides the "Rosetta Stone" for these modern, hybrid cache objects.

Methods

The authors propose a two-stage translation pipeline:

  1. Stage 1 (Linear Alignment): They fit per-layer linear maps in closed form using a few hundred thousand tokens of paired data.
  2. Stage 2 (KL Distillation): They fine-tune the translator to minimize the Kullback–Leibler (KL) divergence between the target model's output using a translated cache versus its own native cache.

Per-Object Recipes

The core contribution is a set of specific strategies for different cache types:

  • MLA Latents: Mapped token-by-token using linear layers. Crucially, they capture the data before positional rotation (RoPE) to keep the map position-independent.
  • Recurrent States: If the source and target have the same state shape, they use a "copy + bilinear correction" method. If the source lacks states entirely, they "replay" the target's own recurrence over predicted per-token inputs.
  • Token Identity: The authors discovered that a linear map alone isn't enough for perfect long-range recall. They decode token IDs directly from the source's first MLA layer (which acts as a deterministic signature of the token) and feed these IDs back into the translation maps.
Figure 1: Switching models mid-conversation without re-reading the prompt. Model S has read the prompt; a translator maps S’s cache (per-token K/V or MLA latents and per-prompt recurrent states) into T’s format, and T answers. Inset: Pair B at 32K tokens, time to T’s first token, cold re-read vs switch (6,823 vs 208 ms, ×32.8).

Datasets

  • Training/Validation: The models were trained and validated on chat conversations from the Nemotron-SFT-Instruction-Following-Chat-v2 dataset.
  • Evaluation Probes:
    • CoQA: Conversational Question Answering.
  • ARC-Challenge: Hard-set reasoning questions.
  • Needle-in-a-Haystack: Testing the model's ability to recall a specific fact buried in up to 128K tokens of context.
  • MT-Bench: Using a "judge" model (Gemma-2-31B) to rate chat quality.

Key Insights and Ideas

  • The MLA "Inversion" Trick: One of the most elegant findings is that the first layer of an MLA model effectively stores the token's identity. By "reading back" the token IDs from the cache, the translator can provide the target model with an exact signal, boosting "needle" recall from a dismal 0.09 to a near-perfect 0.99.
  • Capture Points are Critical: You can't just translate the final tensor the engine stores. To achieve high fidelity, you must capture the "raw" projection outputs before they are normalized or rotated by positional embeddings.
  • Segment-wise Multi-turn: Naively re-translating the whole history at every turn causes quality to collapse. Instead, translating only the new segment and appending it to the existing translated cache maintains "native" performance over long dialogues.

Key Results

The researchers tested their methods on several model pairs, including Kimi-VL-A3B ↔ Kimi-Linear-48B and Qwen3.5-27B ↔ Qwen3.8-27B.

  • Speed: At 64K tokens, the "switch" time was 18.7x to 46.8x faster than a standard re-read. For a Qwen pair, the time to first token dropped from 6.8 seconds to just 208 milliseconds.
  • Quality: In the "Hybrid → Full-MLA" direction (e.g., handing off from a fast hybrid to a flagship), the translated model was statistically indistinguishable from the native model on every task metric.
  • Efficiency: MLA latents are significantly smaller than standard GQA caches, making the transfer even more efficient.
A summary of the recipes used for different cache objects and the evidence supporting them.

Limitations and Future Work

  • Scale: While tested on models up to 48B parameters, the authors state that feasibility for 300B+ "flagship" models remains a hypothesis (though a strongly supported one).
  • Tokenizer Dependency: Currently, the source and target models must share the same tokenizer. Cross-family transfer (e.g., DeepSeek to Kimi) would require an additional "token aligner" layer.
  • Behavioral Leakage: The authors noted that translated states can sometimes carry over the "personality" or safety refusals of the source model—a phenomenon they call behavioral leakage.
  • Sparse Attention: The work on sparse-attention indexers was a "rehearsal" at a smaller scale and needs further validation on massive production models like DeepSeek-V3.

Summary updated

Abstract

A model that takes over another model's conversation must normally re-read the whole history; a text hand-off shortens the re-read only by dropping content. KV-cache translation instead maps the source's cache into the target's own format, so the target continues from the full context without re-reading it. It works for grouped-query attention (GQA), but open-weights frontier models now cache compressed latents (MLA), sparse-attention indexer keys and fixed-size linear-attention states. We replicate KV-Lingo on a GQA pair and then give, for each of these objects, where to capture and inject it and a mechanism that reaches native quality. We study two hybrid pairs at 16-48B scale in both directions, Kimi-VL-A3B - Kimi-Linear-48B (MLA - MLA + Kimi Delta Attention) and Qwen3.5-27B - Qwen3.8-27B (Gated DeltaNet hybrids), plus a sparse-attention target (Qwen3.8-27B - Qwen3.8-Flash-Next). MLA latents translate with token-wise linear maps; recurrent states are copied with a closed-form correction or synthesised by replaying the target's own recurrence. Exact long-range recall needs token identity: decoding token ids from the source's layer-0 MLA latent raises needle recall from 0.09 to 0.99 at 2-32K and to 0.83-1.00 at 64K, eight times the longest training prompt. The hybrid full-MLA direction is within the paired confidence interval of native quality on every task metric, and same-shape GDN hybrids reach the native level in both directions (apart from 6% refusals inherited from the source), with time to first token 18.7 and 46.8 times shorter than a re-read at 64K tokens. Segment-wise multi-turn transfer stays at the re-prefill level; naive re-translation collapses. On the sparse-attention target, predicted indexer keys recall the needle as well as the target's own keys (one seed). We state as a hypothesis how the recipes transfer to 300B+ hybrid flagships.
BibTeX
@misc{kvedaras2026translating,
  author = {Povilas Kvedaras and Tomas Rasymas},
  title = {{Translating KV Caches Between Hybrid-Attention Language Models: MLA Latents, Linear-Attention States and Sparse-Attention Indexers}},
  year = {2026},
  eprint = {2611.12062},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12062},
}
Figure