LiquidCP: Decoupling Attention Tasks from Communication and Kernel Boundaries in Context Parallelism

Hongyu Chen, Yikang Ruan, Yuhai Long, Ziyu Huang, Yiyu Luo, Liukun Yu, Yangjie Zhou, Zihan Liu, Weihao Cui, Zijun Li, Anbang Wu, Yu Feng, Jing Yang, Jingwen Leng, Bingsheng He, Minyi Guo, Shixuan SunarXiv 2026 · arXiv

TL;DR

LiquidCP is a unified context parallelism (CP) framework that decouples attention tasks from rigid communication and kernel boundaries. By treating computation, communication, and reduction as fine-grained, interdependent tasks managed by a persistent GPU engine, it achieves up to $1.35\times$ speedups in LLM training and inference on H100 GPUs.

What Problem Does This Paper Address?

As Large Language Models (LLMs) push toward "infinite" context windows, Context Parallelism (CP)—partitioning sequences across multiple GPUs—has become essential. However, existing CP implementations suffer from two major inefficiencies:

  1. Rigid Execution Boundaries: Traditional "Ring Attention" and "Flash Decoding" frameworks execute in lock-step stages (e.g., wait for a full communication block, then run attention, then reduce). This creates "tail idle" time where many GPU cores sit idle waiting for the slowest task in a stage to finish.
  2. Imbalanced Workloads: In training, standard "zigzag" placement balances the quadratic work of attention but often leaves linear token-wise operations (like LayerNorm or MLP) unevenly distributed. In inference, mixed batches of "prefill" (new prompts) and "decode" (generating tokens) create highly heterogeneous workloads that are difficult to schedule efficiently.

Methods

LiquidCP introduces a common task model where all operations are broken down into three primitives: Transfer, Compute, and Reduction. These are linked by tile-level dependencies rather than kernel boundaries.

MegaRing (for Training and Full Prefill)

For long-sequence training, LiquidCP uses MegaRing, which features:

  • BuddyRing Placement: A novel algorithm that searches for optimal GPU group assignments to balance both attention-work (quadratic) and token-load (linear). It prioritizes keeping short sequences local to avoid fragmentation.
  • Readiness-Driven Execution: Instead of waiting for a full ring-communication step, the system dynamically coalesces ready KV (Key-Value) tiles as they arrive, launching computation as soon as a "readiness threshold" is met.

MegaDCP (for Decoding and Mixed Batches)

For inference, LiquidCP uses MegaDCP, which features:

  • Critical-Wave KV Splitting: A simulator-based model that predicts the completion time of attention and its dependent reductions. It decides how to split history KV shards to maximize parallelism while minimizing the overhead of merging partial results.
  • Release-Aware Ordering: It prioritizes query transfers that unlock the most downstream attention work.

The Persistent Engine

Both designs are executed by a shared, GPU-resident "megakernel." This engine stays active on the GPU, managing task queues and readiness flags through device-side protocols, which bypasses the high latency of frequent host-side kernel launches.

Datasets

The authors evaluated LiquidCP using several diverse workloads:

  • Training/Prefill: Batches sampled from ProLong, ArXiv, FreeLaw, GitHub, and Pile-CC, covering a wide range of sequence length distributions.
  • Inference: 300 snapshots derived from the Mooncake trace (real-world LLM serving logs).
  • End-to-End: A Qwen3-30B-A3B Transformer layer was used to measure total system impact.

Key Insights and Ideas

  • Decoupling is Efficiency: The "aha moment" is that communication steps shouldn't define computation boundaries. By allowing tasks to proceed as soon as their specific data tiles are ready, LiquidCP effectively hides communication latency.
  • The "Tail Idle" Problem: Figure 4 illustrates how serialized stages in standard DCP lead to massive underutilization (up to 48% idle time in mixed batches). LiquidCP’s fine-grained scheduling allows different stages to overlap on the same GPU.
Figure 4: Streaming multiprocessor (SM) activity across seri- alized DCP stages. Timelines show a selected rank for decode- only (top) and mixed (bottom) batches on eight GPUs with TP = 8 and DCP = 2. Colors indicate stage work; gray indi- cates tail idle. The horizontal axis concatenates each stage’s work span, from its earliest start to its latest completion.
  • Dual-Load Balancing: Balancing attention work alone is insufficient for modern LLM layers where non-attention operations are significant. BuddyRing’s ability to satisfy both token and attention tolerances is a key differentiator.

Key Results

LiquidCP demonstrates significant performance gains over state-of-the-art baselines like Megatron-LM, vLLM, and SGLang:

  • Training Speedups: MegaRing achieved geometric-mean core-attention speedups of $1.23\times$ (forward) and $1.18\times$ (backward).
  • Inference Speedups: MegaDCP achieved $1.20\times$ speedups for decode-only batches and $1.35\times$ for mixed batches.
  • End-to-End Serving: Integrated into vLLM, it improved Time-to-First-Token (TTFT) by an average of $1.22\times$.
Figure 1: Core-attention throughput on eight H100 GPUs. Throughput is normalized to the best evaluated baseline in each configuration (dashed line); labels show LiquidCP’s speedups.

Limitations and Future Work

  • Backward Pass Coalescing: While the forward pass benefits from coalescing KV tiles, the authors noted that doing so in the backward pass increases register pressure significantly. They left optimized backward coalescing for future work.
  • Hardware Specificity: The evaluation focused on NVIDIA H100 GPUs. While the principles are general, the specific tile sizes and thresholds ($\tau_{min}, \tau_{max}$) are currently calibrated for this architecture.
  • Host-Side Planning: Although the planners are very fast (under 3ms), they still run on the CPU. Moving the BuddyRing and Critical-Wave planners entirely to the GPU could further reduce overhead in extremely high-throughput scenarios.

Summary updated

Abstract

Context parallelism (CP) partitions long sequences across GPUs for LLM training and inference. For variable-length training and full prefill, attention-balanced placement can leave token loads uneven, while ring execution repeats overhead at each computation and communication step. In stage-serialized decode CP (DCP), long-running tasks delay dependent work and leave substantial GPU hardware idle. We propose LiquidCP, which unifies training and inference through a common model of Transfer, Compute, and Reduction tasks with tile-level dependencies, decoupling attention tasks from communication and kernel boundaries. For training and full prefill, MegaRing incorporates BuddyRing placement to balance attention and token loads, and coalesces ready KV regions during forward attention to amortize overhead. For decoding and mixed batches, MegaDCP selects history-KV splits by jointly modeling the completion of attention and its dependent reductions. A shared persistent GPU-resident engine executes both designs. Compared with the fastest baseline per configuration on eight H100 GPUs, MegaRing achieves geometric-mean core-attention speedups of $1.23\times$ forward and $1.18\times$ backward, and accelerates a Transformer layer by $1.08$--$1.16\times$. MegaDCP achieves geometric-mean core-attention speedups of $1.20\times$ for decode-only and $1.35\times$ for mixed batches. In vLLM serving with KV prefix cache, it achieves an average time-to-first-token speedup of $1.22\times$.
BibTeX
@misc{chen2026liquidcp,
  author = {Hongyu Chen and Yikang Ruan and Yuhai Long and Ziyu Huang and Yiyu Luo and Liukun Yu and Yangjie Zhou and Zihan Liu and Weihao Cui and Zijun Li and Anbang Wu and Yu Feng and Jing Yang and Jingwen Leng and Bingsheng He and Minyi Guo and Shixuan Sun},
  title = {{LiquidCP: Decoupling Attention Tasks from Communication and Kernel Boundaries in Context Parallelism}},
  year = {2026},
  eprint = {2611.04401},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04401},
}
Figure