WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training

Zheng Wang, Anna Cai, Xinfeng Xie, Zaifeng Pan, Yue Guan, Weiwei Chu, Jie Wang, Shikai Li, Jianyu Huang, Chris Cai, Yuchen Hao, Yufei Ding

TL;DR

Large-scale LLM training frequently suffers from GPU underutilization caused by workload imbalance—in Meta’s 8K-GPU LLaMA-3 405B training run, the slowest GPU took up to 1.44× longer to compute than the fastest. WLB-LLM resolves this by introducing workload-aware variable-length packing with outlier document delay at the pipeline parallelism level, alongside adaptive per-document sharding at the context parallelism level. Across diverse model scales, WLB-LLM achieves an average speedup of 1.23× without degrading model convergence.

What Problem Does This Paper Address?

Modern frontier LLMs rely on 4D parallelism—combining Data (DP), Pipeline (PP), Context (CP), and Tensor (TP) Parallelism—to scale across thousands of GPUs. To avoid zero-padding waste during training, multiple documents are packed into uniform-length sequences, using attention masks to prevent cross-document attention.

However, attention computation scales quadratically ($\mathcal{O}(L^2)$) with document length, while other operations (GEMM, LayerNorm, communication) scale linearly ($\mathcal{O}(L)$). A micro-batch packed with a single 128K document demands vastly more compute than one packed with dozens of short documents totaling 128K tokens. Furthermore, standard CP evenly slices sequences across workers, leaving workers assigned the tail ends of long documents with disproportionately heavy workloads. Because workers synchronize via collective communication and pipeline stages, the slowest GPU forces all others to wait, causing severe cluster-wide straggler effects.

Methods

WLB-LLM introduces targeted optimizations at both the PP and CP levels:

  1. Variable-Length Packing (PP Level): Instead of enforcing rigid, fixed-length sequences, WLB-LLM allows micro-batches to vary in sequence length. Shorter documents are packed into slightly longer sequences so their linear operations (GEMMs, activations, communications) compensate for the lower quadratic attention workload, equalizing total micro-batch execution time.
  2. Multi-Level Outlier Delay Queue (PP Level): Extremely long documents cannot be balanced within a single global batch. WLB-LLM temporarily holds outlier documents in a multi-level FIFO queue until enough accumulate to distribute one per micro-batch. Because very long documents represent a tiny fraction of total tokens, tokens are delayed by an average of only 0.5 iterations, preserving dataset randomness and training convergence.
  3. Fine-Grained Per-Document Sharding (CP Level): Rather than blindly partitioning the whole packed sequence, WLB-LLM shards each document into symmetric pairs across CP workers. To avoid padding when document lengths are not multiples of the chunk size, residual tokens are distributed in a round-robin manner.
  4. Adaptive CP Sharding Selection: Slicing documents into smaller chunks can harm GPU hardware efficiency (e.g., dropping below the 128-token tile size of FlashAttention or underutilizing NVIDIA Hopper's Tensor Memory Accelerator [TMA] multicast). WLB-LLM uses an analytical performance model at runtime to dynamically select between per-sequence and per-document sharding based on estimated kernel latency.

Datasets

The authors evaluated WLB-LLM using Meta’s internal pretraining dataset with context windows ranging from 32K to 160K tokens. Document lengths in this corpus are highly skewed: while documents reach the full context length, documents shorter than half the context window account for more than 75% of total tokens.

Key Insights and Ideas

  • Non-attention ops can balance attention imbalance: You don't need equal quadratic attention loads to achieve equal latency. Adding extra tokens from short documents increases linear GEMM and communication time, leveling total runtimes across micro-batches.
  • The "outlier rarity" advantage: Only a minuscule fraction of documents approach the maximum context window. Delaying only these extreme outliers across batches avoids the convergence degradation seen when aggressively reshuffling entire datasets.
  • Workload balance vs. kernel efficiency tradeoff: Perfect mathematical load balancing does not guarantee the fastest runtime. Small document chunks waste thread blocks in tiled attention kernels and forfeit hardware optimizations like Hopper TMA multicast. Runtime adaptation is essential.

Key Results

  • End-to-End Speedup: Evaluated on models from 550M to 70B parameters on 32 nodes (256 NVIDIA H100 GPUs), WLB-LLM achieves an average 1.23× speedup over unoptimized 4D parallelism (Plain-4D) and 1.19× over a fixed-length packing baseline (Fixed-4D).
  • Context Window Scaling: Speedup increases as context lengths expand—reaching 1.40× at a 160K context window on the 7B model.
  • Component Contributions: On a 7B model with a 128K context, CP per-document sharding with adaptive selection yields a 1.05× speedup; PP variable-length packing and outlier delay contribute a 1.28× speedup; combined, they deliver a 1.33× speedup.
  • Zero Convergence Impact: Training loss on a 550M model matched single-batch training curves identically, avoiding the 1.6% loss penalty observed when using multi-batch fixed-length packing windows.

Limitations and Future Work

  • All-or-Nothing CP Decision: Adaptive CP selection currently chooses either per-document or per-sequence sharding for an entire micro-batch. A hybrid strategy—sharding long documents per-document and short documents per-sequence within the same batch—could further boost efficiency.
  • Hardware/Model Scope: Evaluations were conducted on dense transformer architectures on NVIDIA Hopper (H100) GPUs. While the authors discuss theoretical compatibility with Mixture-of-Experts (MoE) and Expert Parallelism (EP), empirical evaluation on MoE workloads remains future work.

Summary updated

Abstract

In this work, we present WLB-LLM, a workLoad-balanced 4D parallelism for large language model training. We first thoroughly analyze the workload imbalance issue in LLM training and identify two primary sources of imbalance at the pipeline parallelism and context parallelism levels. Then, to address the imbalance issue, at the pipeline parallelism level, WLB-LLM incorporates a workload-aware variable-length document packing method to balance the computation and communication workload across micro-batches. Additionally, at the context parallelism level, WLB-LLM introduces a novel fine-grained per-document sharding strategy, ensuring each worker within a context parallelism group has an identical workload. Comprehensive experiments under different model scales demonstrate that WLB-LLM significantly mitigates the workload imbalance during 4D parallelism LLM training and achieves an average speedup of 1.23x when applying WLB-LLM in our internal LLM training framework.
BibTeX
@inproceedings{wang2025wlb,
  author = {Zheng Wang and Anna Cai and Xinfeng Xie and Zaifeng Pan and Yue Guan and Weiwei Chu and Jie Wang and Shikai Li and Jianyu Huang and Chris Cai and Yuchen Hao and Yufei Ding},
  title = {{WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training}},
  booktitle = {OSDI},
  year = {2025},
  url = {https://openreview.net/forum?id=AbqsVJz0tm},
}

Loading related content...

Figure