TL;DR
LiquidCP is a unified context parallelism (CP) framework that decouples attention tasks from rigid communication and kernel boundaries. By treating computation, communication, and reduction as fine-grained, interdependent tasks managed by a persistent GPU engine, it achieves up to $1.35\times$ speedups in LLM training and inference on H100 GPUs.
What Problem Does This Paper Address?
As Large Language Models (LLMs) push toward "infinite" context windows, Context Parallelism (CP)—partitioning sequences across multiple GPUs—has become essential. However, existing CP implementations suffer from two major inefficiencies:
- Rigid Execution Boundaries: Traditional "Ring Attention" and "Flash Decoding" frameworks execute in lock-step stages (e.g., wait for a full communication block, then run attention, then reduce). This creates "tail idle" time where many GPU cores sit idle waiting for the slowest task in a stage to finish.
- Imbalanced Workloads: In training, standard "zigzag" placement balances the quadratic work of attention but often leaves linear token-wise operations (like LayerNorm or MLP) unevenly distributed. In inference, mixed batches of "prefill" (new prompts) and "decode" (generating tokens) create highly heterogeneous workloads that are difficult to schedule efficiently.
Methods
LiquidCP introduces a common task model where all operations are broken down into three primitives: Transfer, Compute, and Reduction. These are linked by tile-level dependencies rather than kernel boundaries.
MegaRing (for Training and Full Prefill)
For long-sequence training, LiquidCP uses MegaRing, which features:
- BuddyRing Placement: A novel algorithm that searches for optimal GPU group assignments to balance both attention-work (quadratic) and token-load (linear). It prioritizes keeping short sequences local to avoid fragmentation.
- Readiness-Driven Execution: Instead of waiting for a full ring-communication step, the system dynamically coalesces ready KV (Key-Value) tiles as they arrive, launching computation as soon as a "readiness threshold" is met.
MegaDCP (for Decoding and Mixed Batches)
For inference, LiquidCP uses MegaDCP, which features:
- Critical-Wave KV Splitting: A simulator-based model that predicts the completion time of attention and its dependent reductions. It decides how to split history KV shards to maximize parallelism while minimizing the overhead of merging partial results.
- Release-Aware Ordering: It prioritizes query transfers that unlock the most downstream attention work.
The Persistent Engine
Both designs are executed by a shared, GPU-resident "megakernel." This engine stays active on the GPU, managing task queues and readiness flags through device-side protocols, which bypasses the high latency of frequent host-side kernel launches.
Datasets
The authors evaluated LiquidCP using several diverse workloads:
- Training/Prefill: Batches sampled from ProLong, ArXiv, FreeLaw, GitHub, and Pile-CC, covering a wide range of sequence length distributions.
- Inference: 300 snapshots derived from the Mooncake trace (real-world LLM serving logs).
- End-to-End: A Qwen3-30B-A3B Transformer layer was used to measure total system impact.
Key Insights and Ideas
- Decoupling is Efficiency: The "aha moment" is that communication steps shouldn't define computation boundaries. By allowing tasks to proceed as soon as their specific data tiles are ready, LiquidCP effectively hides communication latency.
- The "Tail Idle" Problem: Figure 4 illustrates how serialized stages in standard DCP lead to massive underutilization (up to 48% idle time in mixed batches). LiquidCP’s fine-grained scheduling allows different stages to overlap on the same GPU.
- Dual-Load Balancing: Balancing attention work alone is insufficient for modern LLM layers where non-attention operations are significant. BuddyRing’s ability to satisfy both token and attention tolerances is a key differentiator.
Key Results
LiquidCP demonstrates significant performance gains over state-of-the-art baselines like Megatron-LM, vLLM, and SGLang:
- Training Speedups: MegaRing achieved geometric-mean core-attention speedups of $1.23\times$ (forward) and $1.18\times$ (backward).
- Inference Speedups: MegaDCP achieved $1.20\times$ speedups for decode-only batches and $1.35\times$ for mixed batches.
- End-to-End Serving: Integrated into vLLM, it improved Time-to-First-Token (TTFT) by an average of $1.22\times$.
Limitations and Future Work
- Backward Pass Coalescing: While the forward pass benefits from coalescing KV tiles, the authors noted that doing so in the backward pass increases register pressure significantly. They left optimized backward coalescing for future work.
- Hardware Specificity: The evaluation focused on NVIDIA H100 GPUs. While the principles are general, the specific tile sizes and thresholds ($\tau_{min}, \tau_{max}$) are currently calibrated for this architecture.
- Host-Side Planning: Although the planners are very fast (under 3ms), they still run on the CPU. Moving the BuddyRing and Critical-Wave planners entirely to the GPU could further reduce overhead in extremely high-throughput scenarios.
Summary updated