SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference

Yun He, Bin Zhao, Zheng CaoarXiv 2025 · DOI · arXiv

TL;DR

SiPipe is a novel inference framework that significantly boosts Large Language Model (LLM) throughput by offloading auxiliary tasks—like sampling and communication management—to underutilized CPUs. By eliminating the "execution bubbles" (idle time) inherent in pipeline parallelism, SiPipe achieves up to 2.1× higher throughput and 42.7% lower latency compared to state-of-the-art systems like vLLM.

What Problem Does This Paper Address?

When serving massive LLMs (like DeepSeek-V3 or Llama-3.1-405B), a single GPU isn't enough. Engineers use Pipeline Parallelism (PP) to split model layers across multiple GPUs and nodes. However, PP is notoriously inefficient due to "bubbles"—periods where a GPU sits idle waiting for data or instructions.

The authors identify three primary types of bubbles:

  1. Load-Imbalance Bubbles: The final stage of a pipeline usually handles "sampling" (choosing the next word), making it slower than other stages and forcing them to wait.
  2. Intra-Stage Bubbles: GPUs often stall while the CPU prepares input tensors for the next step.
  3. Inter-Stage Bubbles: Moving data between GPUs requires complex "handshakes" to exchange metadata (like tensor shapes), creating synchronization delays.

While GPUs are often pinned at 100% utilization, the host CPUs are typically idling at less than 10%. SiPipe asks: Why not use that "CPU slack" to pop these bubbles?

Methods

SiPipe introduces a heterogeneous design that rebalances the workload between the CPU and GPU through three core techniques:

  • Column-Wise CPU Sampling: Instead of making the last GPU stage handle the computationally sequential task of sampling, SiPipe offloads this to the CPU. It uses a specialized "column-wise" memory layout to make CPU-based penalty calculations and top-k filtering extremely fast and cache-friendly.
  • Token-Safe Execution Model (TSEM): To stop the GPU from waiting on the CPU for input preparation, SiPipe decouples them. It uses a "shadow buffering" scheme with versioned CUDA graphs. This allows the CPU to prepare inputs for iteration N+1 while the GPU is still processing iteration N, without risking data corruption (Write-After-Read hazards).
  • Structure-Aware Transmission (SAT): Standard systems spend 2–5ms just "talking" about the data they are about to send. SAT realizes that while the values in an LLM change, the structure (tensor shapes and types) is mostly stable. By "capturing" this structure once, SiPipe allows GPUs to send data asynchronously, reducing communication overhead to less than 0.1ms.

Datasets

The authors evaluated SiPipe using a variety of industry-leading LLMs and real-world prompt data:

  • Models: Llama-3.1 (70B and 405B), Qwen-2.5-72B, Mixtral-8x7B, DeepSeek-V2.5, and DeepSeek-V3.
  • Workload: Prompts were randomly sampled from the ShareGPT dataset to simulate realistic user interactions.
  • Hardware: Testing was conducted on high-end clusters including H100-NVL, A100-NVL, and A100-PCIe nodes.

Key Insights and Ideas

  • The CPU is an Untapped Resource: In the race for GPU acceleration, we've ignored the fact that modern servers have massive multi-core CPUs that are perfectly capable of handling the "bookkeeping" and sequential logic of LLM inference.
  • Predictability is Power: The "Structure-Aware" insight is crucial. Because LLM architectures are static during inference, we don't need to treat every data transfer as a surprise. Predicting the layout allows for "zero-copy" style communication.
  • Scalability vs. Efficiency: The paper proves that while Tensor Parallelism (TP) is fast for small groups of GPUs, Pipeline Parallelism (PP) is the only viable way to scale across nodes—provided you can solve the bubble problem.

Key Results

  • Throughput: SiPipe delivers 1.6× to 2.1× higher throughput than vLLM on single-node setups and up to 4.5× improvement over pure Tensor Parallelism in cross-node (16-GPU) configurations.
  • Latency: Per-token latency (TPOT) was reduced by up to 43% for ultra-large models.
  • Utilization: Average GPU utilization increased by up to 23%, effectively getting more "work" out of the same expensive hardware.
  • Ablation: CPU sampling provided the biggest boost (up to 38% acceleration), followed closely by the TSEM model.

Limitations and Future Work

  • Python GIL: Because SiPipe is implemented in Python, the authors had to use a multi-process architecture to bypass the Global Interpreter Lock (GIL). They suggest that future versions using Python 3.13’s "free-threaded" mode could be even simpler.
  • MoE Asymmetry: While SiPipe fixes general pipeline bubbles, Mixture-of-Expert (MoE) models have intrinsic workload imbalances (some experts are busier than others). The authors note that MoE-specific optimizations are a separate, complementary challenge.
  • Prefill vs. Decoding: The paper focuses primarily on the decoding phase (generating tokens one by one). While this is the bottleneck for long-running chats, the "prefill" phase (processing the initial prompt) has different characteristics that SiPipe doesn't yet target.

Summary updated

Abstract

As inference workloads for large language models (LLMs) scale to meet growing user demand, pipeline parallelism (PP) has become a widely adopted strategy for multi-GPU deployment, particularly in cross-node setups, to improve key-value (KV) cache capacity and inference throughput. However, PP suffers from inherent inefficiencies caused by three types of execution bubbles-load-imbalance, intra-stage, and inter-stage-which limit pipeline saturation. We present SiPipe, a heterogeneous pipeline design that improves throughput by leveraging underutilized CPU resources to offload auxiliary computation and communication. SiPipe incorporates three key techniques-CPU sampling, a token-safe execution model, and structure-aware transmission-to mitigate pipeline bubbles and improve execution efficiency. Across diverse LLMs, SiPipe achieves up to 2.1 times higher throughput, 43% lower per-token latency, and up to 23% higher average GPU utilization compared to the state-of-the-art vLLM under the same PP configuration, demonstrating its generality across LLMs and deployment scenarios.
BibTeX
@misc{he2025sipipe,
  author = {Yun He and Bin Zhao and Zheng Cao},
  title = {{SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference}},
  year = {2025},
  eprint = {2506.22033},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.22033},
  howpublished = {arXiv},
}
Figure