WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training

Zheng Wang, Anna Cai, Xinfeng Xie, Zaifeng Pan, Yue Guan, Weiwei Chu, Jie Wang, Shikai Li, Jianyu Huang, Chris Cai, Yuchen Hao, Yufei Ding

Abstract

In this work, we present WLB-LLM, a workload-balanced 4D parallelism framework for large language model (LLM) training. We identify two primary sources of workload imbalance at the pipeline parallelism (PP) and context parallelism (CP) levels. To address these, WLB-LLM incorporates a workload-aware variable-length document packing method to balance computation and communication across micro-batches at the PP level. Additionally, at the CP level, it introduces a novel fine-grained per-document sharding strategy to ensure identical workloads for workers within a context parallelism group. Comprehensive experiments across various model scales demonstrate that WLB-LLM significantly mitigates imbalance, achieving an average speedup of 1.23× in internal training frameworks.

Introduction

Training state-of-the-art Large Language Models (LLMs) like Llama 3-405B requires massive computational resources, typically orchestrated through 4D parallelism—a sophisticated combination of Data, Pipeline, Context, and Tensor parallelism. However, the efficiency of these massive clusters is often bottlenecked by the "slowest GPU" problem. In large-scale training jobs, the slowest GPU can take up to 1.44× longer to complete its task than its peers, leading to significant idle time across the rest of the cluster.

This imbalance stems from a fundamental misunderstanding in current training frameworks: the assumption that an equal number of tokens equates to an equal amount of work. Due to the quadratic nature of self-attention, tokens at the end of a long document require significantly more computation than those at the beginning. When documents of varying lengths are packed together, GPUs assigned "heavier" sequences become bottlenecks.

Addressing Imbalance in 4D Parallelism

WLB-LLM introduces targeted optimizations at two critical levels of the parallelism hierarchy to restore computational equilibrium.

1. Pipeline Parallelism: Workload-Aware Variable-Length Packing

Standard packing methods often force fixed sequence lengths, ignoring the arithmetic intensity of the underlying data. WLB-LLM replaces this with a workload-aware approach:

  • Variable-Length Packing: Using a heuristic algorithm, the framework packs documents based on their actual computational cost rather than simple token counts.
  • Outlier Document Delay: Extremely long documents (e.g., 128k tokens) are the primary drivers of imbalance. WLB-LLM utilizes a multi-level waiting queue to strategically delay these "outlier" documents, pairing them with shorter documents in later micro-batches to equalize the total workload across the pipeline.

2. Context Parallelism: Fine-Grained Adaptive Sharding

Traditional Context Parallelism (CP) shards sequences equally across workers. However, if a sequence contains multiple documents of different lengths, the workload remains uneven. WLB-LLM introduces:

  • Per-Document Sharding: Instead of sharding the entire sequence as a single block, WLB-LLM shards each document individually. This ensures that every worker within a CP group receives a symmetrical portion of the computational load.
  • Adaptive Selection: To maintain hardware efficiency, the framework avoids over-sharding. Splitting documents into fragments smaller than a GPU's tile size (typically 128 tokens) can degrade kernel performance. WLB-LLM uses offline profiling to predict kernel latency and dynamically chooses between per-sequence and per-document sharding at runtime.

Experimental Results

The researchers evaluated WLB-LLM using internal Meta training datasets with a 128K context window, testing Llama-like models ranging from 550M to 70B parameters on a cluster of 256 NVIDIA H100 GPUs (32 nodes).

Performance and Efficiency

  • Speedup: WLB-LLM achieved an average end-to-end speedup of 1.23×. For specific configurations, such as the 550M model, speedups reached up to 1.41×.
  • Imbalance Mitigation: The "imbalance degree"—the ratio of maximum latency to average latency—was reduced from 1.44 in the baseline to 1.05 with WLB-LLM.
  • Overhead: The heuristic packing algorithm is highly efficient, adding only ~20ms of overhead per batch, representing less than 0.65% of the total training step time.

Model Convergence

Crucially, the "Outlier Document Delay" strategy did not negatively impact model quality. Training loss curves for WLB-LLM remained nearly identical to the baseline, confirming that the slight reduction in data randomness required for balancing does not hinder the model's ability to learn.

Conclusion

WLB-LLM demonstrates that addressing the heterogeneity of token-level computation is vital for scaling LLM training. By implementing workload-aware packing and adaptive sharding, the framework effectively eliminates the "tail complexity" that plagues large-scale clusters. Future research may extend these balancing techniques to Mixture-of-Experts (MoE) models and deeper integration with GPU communication schedulers.

Summary updated

Abstract

In this work, we present WLB-LLM, a workLoad-balanced 4D parallelism for large language model training. We first thoroughly analyze the workload imbalance issue in LLM training and identify two primary sources of imbalance at the pipeline parallelism and context parallelism levels. Then, to address the imbalance issue, at the pipeline parallelism level, WLB-LLM incorporates a workload-aware variable-length document packing method to balance the computation and communication workload across micro-batches. Additionally, at the context parallelism level, WLB-LLM introduces a novel fine-grained per-document sharding strategy, ensuring each worker within a context parallelism group has an identical workload. Comprehensive experiments under different model scales demonstrate that WLB-LLM significantly mitigates the workload imbalance during 4D parallelism LLM training and achieves an average speedup of 1.23x when applying WLB-LLM in our internal LLM training framework.
BibTeX
@inproceedings{wang2025wlb,
  author = {Zheng Wang and Anna Cai and Xinfeng Xie and Zaifeng Pan and Yue Guan and Weiwei Chu and Jie Wang and Shikai Li and Jianyu Huang and Chris Cai and Yuchen Hao and Yufei Ding},
  title = {{WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training}},
  booktitle = {OSDI},
  year = {2025},
  url = {https://openreview.net/forum?id=AbqsVJz0tm},
}
Figure