MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

Haibin Lin, Ziheng Jiang, Yinmin Zhong

Abstract

MegaScale is a production-grade system designed by ByteDance and Peking University to train Large Language Models (LLMs) on more than 10,000 GPUs. By co-designing algorithms and system architecture, MegaScale achieves a remarkable 55.2% Model FLOPs Utilization (MFU)—a 1.34× improvement over Megatron-LM—while maintaining high stability through advanced automated fault recovery and deep observability tools.

Introduction: The Scaling Walls

Training models with hundreds of billions of parameters is a massive systems engineering challenge. When scaling to over 10,000 GPUs, two primary "walls" emerge:

  1. The Efficiency Wall: Communication overhead often grows faster than computation savings as more GPUs are added. Standard frameworks often see MFU plummet at this scale.
  2. The Stability Wall: In a cluster of 10,000+ GPUs, hardware failures and network "flapping" are statistical certainties. A single slow GPU can bottleneck the entire multi-million dollar training run.

MegaScale addresses these by moving away from "black-box" optimization toward a full-stack approach that modifies everything from the transformer block itself to the physical network topology.

Methods: Algorithm-System Co-design

MegaScale's efficiency stems from tight integration between model architecture and system execution.

Parallel Transformer Blocks and Overlapping

Instead of the standard serial formulation, MegaScale executes Attention and MLP layers in parallel. This reduces computation time without degrading model quality. Crucially, they "hide" communication overhead by breaking GEMM (matrix multiplication) kernels into smaller chunks and pipelining them so that communication (e.g., All-Gather, Reduce-Scatter) happens simultaneously with computation.

Overlapping communication with computation via Parallel Transformer Blocks and fused kernels.

3D Parallelism and Pipeline Optimization

The system combines Data, Pipeline, and Tensor parallelism. To minimize "pipeline bubbles" (idle time), MegaScale uses the LAMB optimizer, allowing batch sizes to scale up to 4× without losing accuracy.

System-Level Enhancements

  • Data Pipeline: "Redundant Dataloader Elimination" uses a tree-based approach so GPUs within a machine share a single data loader, preventing disk I/O bottlenecks.
  • Network Tuning: Custom CLOS network topology reduces ECMP hash conflicts, and tuned retransmit timeouts allow for faster recovery from network glitches.

Operational Stability and Observability

At 10,000+ GPUs, observability is paramount. MegaScale uses a lightweight CUDA Event Monitor to identify "MFU drift" in real-time with negligible overhead. This helps pinpoint "stragglers"—nodes performing ~10% slower—which can otherwise tank cluster efficiency.

Fault tolerance is treated as a core feature. A two-stage "Fast Checkpointing" system saves first to host RAM and then asynchronously to HDFS, reducing recovery time from minutes to seconds. During a production run, the system successfully handled over 100 failures automatically.

Key Results

MegaScale demonstrates near-linear scaling and industry-leading efficiency:

  • Efficiency: Training a 175B model on 12,288 GPUs achieved 55.2% MFU, compared to 41.2% for Megatron-LM.
  • Scalability: For a 530B model, MFU remained stable as the GPU count increased from 2,240 to 11,200.
  • Production Success: In real-world runs on multi-trillion tokens, the system maintained over 90% effective training time despite frequent hardware faults.

Conclusion

MegaScale proves that reaching the next frontier of LLM training requires a holistic view of the stack. By fusing algorithmic changes with deep system optimizations, it overcomes the efficiency and stability barriers inherent in massive-scale distributed training. Components of the system are being open-sourced via the veScale project.

Summary updated

Abstract

We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale brings unprecedented challenges to training efficiency and stability. We take a full-stack approach that co-designs the algorithmic and system components across model block and optimizer design, computation and communication overlapping, operator optimization, data pipeline, and network performance tuning. Maintaining high efficiency throughout the training process (i.e., stability) is an important consideration in production given the long extent of LLM training jobs. Many hard stability issues only emerge at large scale, and in-depth observability is the key to address them. We develop a set of diagnosis tools to monitor system components and events deep in the stack, identify root causes, and derive effective techniques to achieve fault tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the MFU by 1.34x compared to Megatron-LM. We share our operational experience in identifying and fixing failures and stragglers. We hope by articulating the problems and sharing our experience from a systems perspective, this work can inspire future LLM systems research.
BibTeX
@inproceedings{lin2024megascale,
  author = {Haibin Lin and Ziheng Jiang and Yinmin Zhong},
  title = {{MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs}},
  booktitle = {NSDI},
  year = {2024},
  url = {https://openreview.net/forum?id=E1YH1NuDD4},
}
Figure