Overcoming the PCIe Bottleneck in Offloaded LLM Inference

This direction is merged into another direction: data transfer latency. Its last version stays readable and citable here.

Version 1 of 1151 papers166 conceptsfrozen 2026-10-04

The primary obstacle to running state-of-the-art Large Language Models (LLMs) on consumer hardware is not a lack of raw computing power, but a lack of Video RAM (VRAM). When a model’s weights or its "memory" of the conversation (the KV cache) exceeds the capacity of a GPU, the standard solution is offloading: storing the data in the computer’s system RAM (CPU memory) and swapping it onto the GPU as needed.

However, this introduces a new "memory wall." The PCIe bus—the physical connection between the CPU and GPU—is orders of magnitude slower than the GPU’s internal memory. In a naive offloading setup, the GPU spends most of its time idle, waiting for data to arrive. Recent research has focused on breaking this bottleneck by ensuring that we only move the most critical data across the bus, or by hiding the transfer time behind useful work.

The KV Cache Wall and Selective Retrieval

For long-context tasks, the Key-Value (KV) cache—which stores the mathematical representations of all previous tokens—can become even larger than the model itself. Yao Fu describes this as a "concurrency cliff," where the memory demands of a single long-context user can paralyze a GPU.

To solve this, researchers are treating the KV cache not as a static block of data to be moved, but as a database to be queried. Di Liu and Chen Meng introduced RetrievalAttention, which uses vector search to identify and retrieve only the most relevant 1–3% of the KV cache from CPU memory. By only transferring the "important" tokens, the system avoids the PCIe bottleneck entirely. Similarly, RetroInfer by Mao Yang uses an "attention-aware" index to maintain accuracy while slashing the volume of data moved across the bus.

Speculation as a Latency Hider

If we must wait for weights to travel across the PCIe bus, we should make every transfer count. Speculative Decoding is a technique where a small, fast "draft" model (resident on the GPU) guesses several future tokens, and the large "target" model (offloaded to the CPU) verifies them all at once.

Avner May developed SpecExec, which builds a massive tree of potential token continuations. This allows the system to validate many tokens in a single parallel pass, effectively amortizing the cost of the weight transfer over multiple generated words. Pei-Shuo Wang and Jian-Jia Chen took this further with SubSpec, which creates a draft model from the target model's own quantized layers, ensuring high alignment and reducing the number of "mispredictions" that cause costly stalls.

Exploiting Activation Sparsity

Not all parts of a neural network are equally important for every prompt. Yixin Song and Zeyu Mi observed in PowerInfer that LLM activations follow a "power-law" distribution: a small subset of "hot" neurons are triggered frequently, while most neurons remain "cold."

By keeping the "hot" neurons on the GPU and only fetching "cold" neurons from the CPU when absolutely necessary, PowerInfer achieves significant speedups. This concept is particularly potent for Mixture-of-Experts (MoE) models, where only a few "experts" are active for any given token. Fiddler and DuoServe-MoE by Yuning Zhang use predictive algorithms to "prefetch" the required experts into GPU memory just before they are needed, attempting to overlap the slow PCIe transfer with the GPU's active computation.

Heterogeneous Orchestration

Finally, some researchers argue that we shouldn't just treat the CPU as a slow storage bin, but as a co-processor. ScheInfer by Wen-Juan Lin and Xiaowen Chu "slices" model weights and parallelizes tasks across the CPU, GPU, and the PCIe bus simultaneously.

By leveraging modern CPU features like Intel’s Advanced Matrix Extensions (AMX), as explored by Hyungyo Kim and Amir Yazdanbakhsh, systems can perform part of the inference directly on the CPU. This reduces the total amount of data that needs to cross the PCIe bus, as the CPU can process its assigned layers and only send the resulting hidden states—which are much smaller than weight matrices—to the GPU.

Papers in this version

Showing 60 of 151 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_overcoming_the_pcie_bottleneck_in_offloa_f111cb1b,
  title = {Overcoming the PCIe Bottleneck in Offloaded LLM Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-pcie-bottleneck-in-offloaded-llm-inference/rdr_f111cb1baffc451faeae98159b1aa4f1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_overcoming_the_pcie_bottleneck_in_offloa_f111cb1b_v1,
  title = {Overcoming the PCIe Bottleneck in Offloaded LLM Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-pcie-bottleneck-in-offloaded-llm-inference/rdr_f111cb1baffc451faeae98159b1aa4f1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}