BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation

Payman Behnam, Debopam Sanyal, Pranav Ponnusamy, Kusum Meda Ravi, Pranav Sastry, Mani Chandana Bandaru, Alind Khare, Myungjin Lee, Alexey TumanovarXiv 2026 · arXiv

Large Language Model (LLM) inference is a tale of two phases. First, the prefill phase processes the entire prompt in parallel—it's compute-heavy but happens only once. Then comes the decoding phase, which generates tokens one by one. This second phase is a notorious bottleneck; it's limited by memory bandwidth because the model must reload all its weights and the growing "KV cache" (the memory of previous tokens) for every single new word.

Researchers from Georgia Tech and Microsoft recently unveiled BridgeKV, a clever "best of both worlds" approach that uses a large, smart model for prefill and a small, fast model for decoding.

TL;DR

BridgeKV enables high-accuracy, low-latency LLM serving by using a large model for the compute-intensive prefill phase and a smaller model for the memory-bound decoding phase. By "stitching" the large model's rich context into the smaller model's format, it achieves up to 3.0× faster requests on a single GPU while recovering up to 80% of the accuracy gap between the two models.

What Problem Does This Paper Address?

Currently, serving systems usually pick one model and stick with it. If you choose a 70B parameter model, you get great answers, but decoding is painfully slow and expensive. If you choose an 8B model, it’s snappy, but it might miss the nuances of a complex prompt.

This "phase asymmetry" suggests a split: use the "smart" model to understand the prompt and the "fast" model to write the response. However, you can't just hand over the KV cache from a large model to a small one. They have different numbers of layers, different internal dimensions, and even different ways of representing token positions (Rotary Position Embeddings, or RoPE). BridgeKV bridges this representational chasm.

Methods

The core of BridgeKV is a lightweight KV-stitching module that translates the large model’s prompt cache into the smaller model’s layout.

Figure 2: Overview of main components in BridgeKV. The left side shows windowed layer mapping while the right side illustrates KV cache stitching and the two training stages. For keys, the RoPE of Mp is removed before the stitching layer and the RoPE of Md is applied after it.

Key technical innovations include:

  • Windowed Layer Mapping: Instead of a rigid one-to-one layer mapping, each layer in the small model learns to aggregate information from a "window" of nearby layers in the large model.
  • Residual Nonlinear Stitching: The module uses a base linear projection (to fix dimensions) plus a small nonlinear MLP branch that learns to correct complex mismatches between the models' latent spaces.
  • RoPE-Aware Translation: Since models "rotate" their keys to encode position, BridgeKV "un-rotates" the large model's keys, performs the stitching in a position-neutral space, and then applies the small model's specific rotation.
  • Two-Stage Training:
    1. Warmup: Both models are frozen. The stitching module is trained to align the caches and use KL-divergence distillation so the small model mimics the large model's next-token predictions.
  • Task Adaptation: If needed, the module and small-model LoRA adapters are jointly fine-tuned on specific datasets.

Datasets

The authors evaluated BridgeKV across six model pairs (from the Llama 3.1/3.2 and Qwen 2.5 families) using five diverse datasets:

  • NarrativeQA: Single-document question answering.
  • CoQA: Conversational QA.
  • SAMSum: Dialogue summarization.
  • GovReport & Multi-News: Long-form summarization.

These tasks were chosen to stress the model's ability to handle long contexts and complex instructions.

Key Insights and Ideas

  • The "Smarts" are in the Cache: A major "aha" moment is that much of a large model's superior performance is baked into how it represents the initial prompt. If you can successfully transfer that "understanding" (the KV cache), a smaller model can often take it from there with minimal quality loss.
  • Linearity isn't Enough: The authors found that simple linear mappings between models fail to capture the nuances of different architectures. The nonlinear residual branch in their stitching module was crucial for maintaining accuracy.
  • Zero-Overhead Decoding: Because the stitching happens only once (at the end of prefill), the per-token decoding speed is exactly as fast as the standalone small model. The "stitching tax" is paid only once per request.

Key Results

BridgeKV consistently outperformed fine-tuned small-model baselines that had the same number of trainable parameters.

Figure 1: Request latency and test accuracy comparison. BridgeKV (Llama 8B -> 1B) achieves latency near the 1B model with accuracy much closer to the 8B model.
  • Speed & Efficiency: For a Llama-3.1-8B to Llama-3.2-1B handoff on an NVIDIA A40, BridgeKV served requests 3.0× faster than the 8B model alone.
  • Accuracy Recovery: It closed up to 80% of the quality gap between the small and large models. On CoQA, a Qwen 7B to 3B handoff reached 81.5 F1, nearly matching the 7B model's 81.9 F1.
  • Massive Gains at Scale: In simulations of "disaggregated" serving (where prefill and decode happen on different GPUs), a Llama 70B to 1B handoff predicted a staggering 33.86× faster response time and 27.5× lower energy use compared to using the 70B model for everything.

Limitations and Future Work

While impressive, BridgeKV has a few constraints:

  • Shared Tokenizer: The prefill and decode models must share the same tokenizer. This currently limits the technique to models within the same family (e.g., Llama to Llama).
  • Static Handoff: The cache is translated once at the start. For extremely long generations, the small model's own "drift" might eventually degrade quality. The authors suggest "richer policies" that adapt the handoff during generation as a future path.
  • Training Required: You still need to train one stitching module per model pair and dataset, though the authors show this is cheaper than standard fine-tuning.

BridgeKV represents a significant shift in how we think about LLM deployment—moving away from "one size fits all" toward a more surgical, phase-aware architecture.

Summary updated

Abstract

LLM inference has two phases with different bottlenecks: LLM prefill runs once per request and is compute-intensive. Decoding repeats for every output token and is limited by memory bandwidth. To address the challenges associated with this issue, different solutions have been proposed. All of these solutions use the same model for both prefill and decode phases. We propose BridgeKV, which uses a larger model for prefill and a smaller model of the same family for decoding. A lightweight KV-stitching module translates the larger model's prompt cache into the smaller model's cache layout. It handles differences in cache shape and representation through learned local layer mixing and separate nonlinear maps for keys and values. With both models frozen, we train the module to align their caches and make the smaller model follow the larger model's next-token predictions. A second training stage adds low-rank adapters when further task adaptation is needed. Across six model pairs and five datasets, BridgeKV outperforms fine-tuned small-model baselines with matched trainable parameter budgets in all 30 settings. It closes up to 80% of the quality gap to using the larger model for both phases. On one A40 GPU, Llama-3.1-Instruct-8B to Llama-3.2-Instruct-1B achieves 3.0X faster requests than the larger model alone, with latency close to the smaller model alone. BridgeKV can improve the hardware and system solutions that separate prefill and decode phases across GPUs. A separate serving simulation models request arrivals, queues, and cache transfer. For Llama-3.1-Instruct-70B to Llama-3.2-Instruct-1B, it predicts up to 33.86X faster requests and 27.5X lower energy use than using the larger model for both prefill and decode phases.
BibTeX
@misc{behnam2026bridgekv,
  author = {Payman Behnam and Debopam Sanyal and Pranav Ponnusamy and Kusum Meda Ravi and Pranav Sastry and Mani Chandana Bandaru and Alind Khare and Myungjin Lee and Alexey Tumanov},
  title = {{BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation}},
  year = {2026},
  eprint = {2611.11761},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11761},
}
Figure