MiMatrix: A Massively Distributed Deep Learning Framework on a Petascale High-density Heterogeneous Cluster

Xin Chen, Hua Zhou, Yuxiang Gao, Yu Zhu, Dongyan Wang

Abstract: In this paper, we present a co-designed petascale high-density GPU cluster to expedite distributed deep learning training with synchronous Stochastic Gradient Descent (SSGD). This architecture of our heterogeneous cluster is inspired by Harvard architecture. Regarding to different roles in the system, nodes are configured as different specifications. Based on the topology of the whole system's network and properties of different types of nodes, we develop and implement a novel job server parallel software framework, named by MiMatrix, for distributed deep learning training. Compared to the parameter server framework, in which parameter server is a bottleneck of data transfer in AllReduce algorithm of SSGD, the job server undertakes all of controlling, scheduling and monitoring tasks without model data transfer. In MiMatrix, we propose a novel GPUDirect Remote direct memory access (RDMA)-aware parallel algorithm of AllReduce executed by computing servers, which both computation and handshake message are $O(1)$ at each epoch.

Introduction

As deep learning models grow in complexity, training them requires massive distributed computing power. Traditionally, this is handled via a Parameter Server architecture, where a central node collects gradients from "workers," averages them, and sends them back. However, this creates two major challenges: bandwidth bottlenecks at the central server and the "stagger problem," where the entire system waits for the slowest worker, leading to poor hardware utilization.

Researchers at the Midea Corporate Research Center addressed these issues by co-designing a custom petascale GPU cluster ("Manoa") alongside a specialized software framework ("MiMatrix"). This full-stack solution achieves state-of-the-art training speeds while significantly undercutting the price of commercial "turnkey" AI supercomputers.

System Architecture and Hardware Co-design

The authors' solution is a "co-design" approach, meaning they built the hardware and software to fit each other perfectly. The Manoa Cluster is inspired by the Harvard architecture, which separates instruction and data memory. In this system, control and data paths are similarly decoupled.

The Manoa cluster network topology, illustrating the separation of the Job Server (control) from the high-speed InfiniBand data network.

The cluster consists of 16 computing nodes linked via high-speed InfiniBand FDR56 (56Gbps) and 10G Ethernet. Each node is a high-density unit packed with 8 NVIDIA Tesla P100 GPUs.

Internal architecture of a computing node, showing the dual-socket Intel Xeon configuration and the PCIe switch fabric connecting 8 GPUs and dual InfiniBand HCAs.

The MiMatrix Framework and GDRAA Algorithm

Instead of a traditional Parameter Server, the MiMatrix Framework utilizes a Job Server. This central node handles scheduling and monitoring but never touches the model weights. By removing model weight transfer from the central node, the Job Server can manage thousands of tasks without becoming a bottleneck.

To facilitate efficient data synchronization between workers, the authors introduced the GPUDirect RDMA-Aware AllReduce (GDRAA) algorithm. This algorithm allows GPUs to communicate directly across the network (Peer-to-Peer) using low-level InfiniBand programming (ibverbs). This bypasses the CPU and system memory entirely, leading to "zero-copy" data transfers. The GDRAA algorithm is mathematically optimized so that the number of "handshake" messages required for synchronization remains constant ($O(1)$) regardless of the number of workers, provided the memory size is fixed.

Experimental Evaluation

The system was evaluated using the industry-standard ImageNet-1K dataset (1.2 million images). The researchers used ResNet-50 and ResNet-101 architectures as benchmarks to measure both accuracy and scaling efficiency.

Key Results

  • Raw Performance: The Manoa cluster delivers over 1.2 PetaFLOPS of single-precision performance.
  • Economic Efficiency: The total hardware cost was approximately $900,000, less than 45% of the cost of an equivalent commercial solution at the time. The authors proposed a new metric, the Price and Convergence Ratio (PCR), to measure the cost-effectiveness of reaching specific accuracy levels.
  • Training Speed: ResNet-101 reached 70% Top-1 accuracy in 888 minutes (approx. 14.8 hours).
  • Scalability: The system demonstrated excellent scaling, achieving a 14.56x speedup when moving from 1 worker to 32 workers (utilizing 128 GPUs).

Conclusion and Future Work

MiMatrix demonstrates that a co-designed, high-density heterogeneous cluster can provide petascale performance at a fraction of the cost of proprietary systems. While the current work focused on hardware and framework efficiency, future research will explore hyperparameter optimization (such as LARS) and testing the $O(1)$ complexity on even larger datasets like ImageNet-22K.

Summary updated

Abstract

In this paper, we present a co-designed petascale high-density GPU cluster to expedite distributed deep learning training with synchronous Stochastic Gradient Descent~(SSGD). This architecture of our heterogeneous cluster is inspired by Harvard architecture. Regarding to different roles in the system, nodes are configured as different specifications. Based on the topology of the whole system's network and properties of different types of nodes, we develop and implement a novel job server parallel software framework, named by \textit{MiMatrix}, for distributed deep learning training. Compared to the parameter server framework, in which parameter server is a bottleneck of data transfer in AllReduce algorithm of SSGD, the job server undertakes all of controlling, scheduling and monitoring tasks without model data transfer. In MiMatrix, we propose a novel GPUDirect Remote direct memory access~(RDMA)-aware parallel algorithm of AllReucde executed by computing servers, which both computation and handshake message are $O(1)$ at each epoch
BibTeX
@misc{chen2018mimatrix,
  author = {Xin Chen and Hua Zhou and Yuxiang Gao and Yu Zhu and Dongyan Wang},
  title = {{MiMatrix: A Massively Distributed Deep Learning Framework on a Petascale High-density Heterogeneous Cluster}},
  year = {2018},
  eprint = {1802.02326},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.02326},
}
Figure