Abstract
MegaScale is a production-grade system designed by ByteDance and Peking University to train Large Language Models (LLMs) on more than 10,000 GPUs. By co-designing algorithms and system architecture, MegaScale achieves a remarkable 55.2% Model FLOPs Utilization (MFU)—a 1.34× improvement over Megatron-LM—while maintaining high stability through advanced automated fault recovery and deep observability tools.
Introduction: The Scaling Walls
Training models with hundreds of billions of parameters is a massive systems engineering challenge. When scaling to over 10,000 GPUs, two primary "walls" emerge:
- The Efficiency Wall: Communication overhead often grows faster than computation savings as more GPUs are added. Standard frameworks often see MFU plummet at this scale.
- The Stability Wall: In a cluster of 10,000+ GPUs, hardware failures and network "flapping" are statistical certainties. A single slow GPU can bottleneck the entire multi-million dollar training run.
MegaScale addresses these by moving away from "black-box" optimization toward a full-stack approach that modifies everything from the transformer block itself to the physical network topology.
Methods: Algorithm-System Co-design
MegaScale's efficiency stems from tight integration between model architecture and system execution.
Parallel Transformer Blocks and Overlapping
Instead of the standard serial formulation, MegaScale executes Attention and MLP layers in parallel. This reduces computation time without degrading model quality. Crucially, they "hide" communication overhead by breaking GEMM (matrix multiplication) kernels into smaller chunks and pipelining them so that communication (e.g., All-Gather, Reduce-Scatter) happens simultaneously with computation.
3D Parallelism and Pipeline Optimization
The system combines Data, Pipeline, and Tensor parallelism. To minimize "pipeline bubbles" (idle time), MegaScale uses the LAMB optimizer, allowing batch sizes to scale up to 4× without losing accuracy.
System-Level Enhancements
- Data Pipeline: "Redundant Dataloader Elimination" uses a tree-based approach so GPUs within a machine share a single data loader, preventing disk I/O bottlenecks.
- Network Tuning: Custom CLOS network topology reduces ECMP hash conflicts, and tuned retransmit timeouts allow for faster recovery from network glitches.
Operational Stability and Observability
At 10,000+ GPUs, observability is paramount. MegaScale uses a lightweight CUDA Event Monitor to identify "MFU drift" in real-time with negligible overhead. This helps pinpoint "stragglers"—nodes performing ~10% slower—which can otherwise tank cluster efficiency.
Fault tolerance is treated as a core feature. A two-stage "Fast Checkpointing" system saves first to host RAM and then asynchronously to HDFS, reducing recovery time from minutes to seconds. During a production run, the system successfully handled over 100 failures automatically.
Key Results
MegaScale demonstrates near-linear scaling and industry-leading efficiency:
- Efficiency: Training a 175B model on 12,288 GPUs achieved 55.2% MFU, compared to 41.2% for Megatron-LM.
- Scalability: For a 530B model, MFU remained stable as the GPU count increased from 2,240 to 11,200.
- Production Success: In real-world runs on multi-trillion tokens, the system maintained over 90% effective training time despite frequent hardware faults.
Conclusion
MegaScale proves that reaching the next frontier of LLM training requires a holistic view of the stack. By fusing algorithmic changes with deep system optimizations, it overcomes the efficiency and stability barriers inherent in massive-scale distributed training. Components of the system are being open-sourced via the veScale project.
Summary updated