MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud

Zhen Zhang, Shuai Zheng, Yida Wang, Justin Chiu, George Karypis, Trishul Chilimbi, Mu Li, Xin Jin

Introduction

The training of "gigantic" AI models—those featuring billions or even trillions of parameters—has become a cornerstone of modern machine learning. However, these models exceed the memory capacity of any single GPU, necessitating distributed training strategies. While existing solutions like Microsoft’s ZeRO-3 partition model states across a cluster, they are often optimized for specialized supercomputers with high-speed, balanced networking. In contrast, public cloud environments present a significant challenge: "unbalanced" networks where intra-node communication (e.g., via NVLink) can be up to 24 times faster than inter-node communication.

This paper introduces MiCS (Minimizing Communication Scale), a distributed training system designed specifically to overcome the communication bottlenecks of public clouds. By reducing the number of participants in communication collectives and leveraging hierarchical network structures, MiCS achieves near-linear scaling efficiency and significantly higher throughput than state-of-the-art systems.

The Communication Bottleneck in Public Clouds

Standard data-parallel frameworks like ZeRO-3 treat all GPUs in a cluster as a single flat group. As the model size and cluster size grow, the "communication scale"—the number of GPUs involved in a single collective operation—increases. In a public cloud, this forces frequent, massive data transfers over relatively slow inter-node links. This leads to high latency and poor bandwidth utilization, preventing the system from scaling efficiently beyond a few dozen GPUs.

The MiCS Approach

To address these inefficiencies, MiCS employs three primary strategies focused on minimizing the scale and cost of communication:

1. Scale-aware Model Partitioning

Rather than partitioning the model across the entire cluster, MiCS partitions it across the smallest possible "partition group" that can fit the model (for example, a single node of 8 GPUs). The system then creates multiple replicas of these partition groups across the cluster. This ensures that the most frequent communication operations, such as parameter gathering, occur over the fastest available links (intra-node NVLink) rather than the slower inter-node network.

2. Hierarchical Communication Strategy

In cases where a model is too large to fit within a single node and must span multiple nodes, MiCS utilizes a three-stage "all-gather" process. This hierarchical approach aggregates data across nodes in parallel, rearranges it, and then performs a batched intra-node synchronization. By structuring communication this way, MiCS reduces the total volume of data that must traverse the slower inter-node connections.

3. 2-hop Gradient Synchronization

Standard distributed training typically triggers a global gradient synchronization after every micro-step. MiCS introduces a "2-hop" approach: it synchronizes gradients only within the local partition group for each micro-step. The expensive global synchronization across all replicas is delayed until the "gradient accumulation boundary" (the end of a full batch). This amortizes the communication cost over multiple computation steps, significantly reducing the overhead of global barriers.

Beyond these core strategies, MiCS incorporates fine-grained synchronization to overlap computation with communication and a custom memory manager to mitigate memory fragmentation during long training runs.

Evaluation and Results

The authors evaluated MiCS using various massive architectures, including BERT (up to 50B parameters), RoBERTa, and GPT-2, as well as WideResNet for computer vision tasks.

Throughput and Scaling

MiCS demonstrated substantial performance improvements over DeepSpeed (ZeRO-3). On AWS V100 clusters, MiCS achieved up to 2.89× higher throughput, and on A100 clusters, it was up to 2.74× faster. Most notably, MiCS showed a 99.4% weak-scaling efficiency when training a 100B parameter model on 512 GPUs, maintaining near-linear performance as the hardware resources increased.

Hardware Utilization

In public cloud environments, which are often restricted by networking limits, MiCS managed to saturate over 54.5% of the theoretical peak compute power of the GPUs. When compared to specialized systems like Megatron-LM-3D (which uses Tensor and Pipeline parallelism), MiCS provided up to 31% higher throughput for standard transformer models. Furthermore, MiCS offers a simpler user experience, as it does not require the complex code refactoring often associated with model-parallel approaches.

Conclusion and Future Work

MiCS proves that by embracing the hierarchical nature of cloud networks and minimizing the scale of communication collectives, it is possible to train gigantic models with extreme efficiency. The "aha moment" of the research is that the number of participants in a communication collective is a critical factor in system performance.

However, some limitations remain. For models with unconventional structures (e.g., extremely wide layers), model-parallel approaches like Megatron-LM may still offer marginal advantages. Additionally, MiCS currently requires users to manually select the "partition group size." Future work will focus on developing automated heuristics to optimize these parameters and improving memory usage prediction within the dynamic PyTorch environment.

Summary updated

BibTeX
@article{zhang2022mics,
  author = {Zhen Zhang and Shuai Zheng and Yida Wang and Justin Chiu and George Karypis and Trishul Chilimbi and Mu Li and Xin Jin},
  title = {{MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud}},
  journal = {Proc. VLDB Endow.},
  year = {2022},
  url = {https://openreview.net/forum?id=4S4bYeTgLz},
}
Figure