Abstract
In this work, we present WLB-LLM, a workload-balanced 4D parallelism framework for large language model (LLM) training. We identify two primary sources of workload imbalance at the pipeline parallelism (PP) and context parallelism (CP) levels. To address these, WLB-LLM incorporates a workload-aware variable-length document packing method to balance computation and communication across micro-batches at the PP level. Additionally, at the CP level, it introduces a novel fine-grained per-document sharding strategy to ensure identical workloads for workers within a context parallelism group. Comprehensive experiments across various model scales demonstrate that WLB-LLM significantly mitigates imbalance, achieving an average speedup of 1.23× in internal training frameworks.
Introduction
Training state-of-the-art Large Language Models (LLMs) like Llama 3-405B requires massive computational resources, typically orchestrated through 4D parallelism—a sophisticated combination of Data, Pipeline, Context, and Tensor parallelism. However, the efficiency of these massive clusters is often bottlenecked by the "slowest GPU" problem. In large-scale training jobs, the slowest GPU can take up to 1.44× longer to complete its task than its peers, leading to significant idle time across the rest of the cluster.
This imbalance stems from a fundamental misunderstanding in current training frameworks: the assumption that an equal number of tokens equates to an equal amount of work. Due to the quadratic nature of self-attention, tokens at the end of a long document require significantly more computation than those at the beginning. When documents of varying lengths are packed together, GPUs assigned "heavier" sequences become bottlenecks.
Addressing Imbalance in 4D Parallelism
WLB-LLM introduces targeted optimizations at two critical levels of the parallelism hierarchy to restore computational equilibrium.
1. Pipeline Parallelism: Workload-Aware Variable-Length Packing
Standard packing methods often force fixed sequence lengths, ignoring the arithmetic intensity of the underlying data. WLB-LLM replaces this with a workload-aware approach:
- Variable-Length Packing: Using a heuristic algorithm, the framework packs documents based on their actual computational cost rather than simple token counts.
- Outlier Document Delay: Extremely long documents (e.g., 128k tokens) are the primary drivers of imbalance. WLB-LLM utilizes a multi-level waiting queue to strategically delay these "outlier" documents, pairing them with shorter documents in later micro-batches to equalize the total workload across the pipeline.
2. Context Parallelism: Fine-Grained Adaptive Sharding
Traditional Context Parallelism (CP) shards sequences equally across workers. However, if a sequence contains multiple documents of different lengths, the workload remains uneven. WLB-LLM introduces:
- Per-Document Sharding: Instead of sharding the entire sequence as a single block, WLB-LLM shards each document individually. This ensures that every worker within a CP group receives a symmetrical portion of the computational load.
- Adaptive Selection: To maintain hardware efficiency, the framework avoids over-sharding. Splitting documents into fragments smaller than a GPU's tile size (typically 128 tokens) can degrade kernel performance. WLB-LLM uses offline profiling to predict kernel latency and dynamically chooses between per-sequence and per-document sharding at runtime.
Experimental Results
The researchers evaluated WLB-LLM using internal Meta training datasets with a 128K context window, testing Llama-like models ranging from 550M to 70B parameters on a cluster of 256 NVIDIA H100 GPUs (32 nodes).
Performance and Efficiency
- Speedup: WLB-LLM achieved an average end-to-end speedup of 1.23×. For specific configurations, such as the 550M model, speedups reached up to 1.41×.
- Imbalance Mitigation: The "imbalance degree"—the ratio of maximum latency to average latency—was reduced from 1.44 in the baseline to 1.05 with WLB-LLM.
- Overhead: The heuristic packing algorithm is highly efficient, adding only ~20ms of overhead per batch, representing less than 0.65% of the total training step time.
Model Convergence
Crucially, the "Outlier Document Delay" strategy did not negatively impact model quality. Training loss curves for WLB-LLM remained nearly identical to the baseline, confirming that the slight reduction in data randomness required for balancing does not hinder the model's ability to learn.
Conclusion
WLB-LLM demonstrates that addressing the heterogeneity of token-level computation is vital for scaling LLM training. By implementing workload-aware packing and adaptive sharding, the framework effectively eliminates the "tail complexity" that plagues large-scale clusters. Future research may extend these balancing techniques to Mixture-of-Experts (MoE) models and deeper integration with GPU communication schedulers.
Summary updated