A Novel Co-design Peta-scale Heterogeneous Cluster for Deep Learning Training

Xin Chen, Hua Zhou, Yuxiang Gao, Yu Zhu

Abstract

Large scale deep Convolutional Neural Networks (CNNs) increasingly demand immense computing power. It is critical for researchers to possess powerful computing platforms to leverage advancements in deep learning (DL). However, commodity GPU accelerators are becoming increasingly expensive. This paper presents a co-designed distributed system including a peta-scale GPU cluster, "Manoa," and a tailored software framework, "MiMatrix." MiMatrix utilizes a job server for scheduling and monitoring while employing a novel AllReduce algorithm, GPUDirect RDMA-Aware AllReduce (GDRAA), to eliminate bandwidth bottlenecks. Our results on ResNet-50 and ResNet-101 benchmarks demonstrate state-of-the-art performance and high cost-efficiency.

1. Introduction

The rapid evolution of deep learning is constrained by the "Scaling Wall" and the "Hardware Dilemma." Traditional distributed training often relies on a central parameter server, which becomes a bottleneck as cluster size increases. Furthermore, the high cost of turnkey solutions limits accessibility for many research teams. We address these challenges through a co-design approach, optimizing both the hardware topology and the communication software to achieve peta-scale performance at a fraction of the market cost.

2. The Manoa Cluster Architecture

The Manoa cluster is a heterogeneous system designed for high-density, high-throughput computation. It delivers over 1.2 PetaFLOPS of single-precision performance within a compact footprint.

2.1 Cluster Topology

The system architecture separates control and data planes. A central Job Server manages scheduling, monitoring, and I/O tasks, while the heavy lifting of data synchronization occurs directly between computing nodes via a high-speed InfiniBand fabric. This prevents the Job Server from becoming a bandwidth bottleneck during weight updates.

Figure 1: Topology of the Manoa cluster showing the separation of the Job Server, Storage, and Computing Servers via InfiniBand and Ethernet switches.

2.2 Computing Node Design

Each of the 16 computing nodes is engineered for maximum internal bandwidth. A single node contains dual Intel Xeon CPUs and eight NVIDIA Tesla GPUs. To facilitate efficient peer-to-peer communication, we utilize PCIe switches and InfiniBand FDR Host Channel Adapters (HCAs), allowing for direct memory access across the network.

Figure 2: Internal architecture of a Manoa computing node, featuring dual CPUs, eight GPUs, and PCIe switching for high-speed local interconnects.

3. The MiMatrix Framework and GDRAA Algorithm

To exploit Manoa's hardware, we developed MiMatrix. Unlike traditional parameter server models, MiMatrix's Job Server does not participate in the weight data transfer for AllReduce processing. The core of MiMatrix is the GPUDirect RDMA-Aware AllReduce (GDRAA) algorithm. GDRAA leverages low-level ibverbs and RDMA to allow GPUs to communicate directly across the network. The algorithm achieves $O(1)$ complexity for both computation and handshake messages relative to the number of workers, with only two synchronizations per iteration—the theoretical minimum.

4. Experimental Evaluation

We benchmarked the system using the ImageNet-1K dataset (1.2 million images).

  • Performance: Training ResNet-101 to 70% top-1 accuracy took 888 minutes (approx. 14.8 hours).
  • Scalability: The system achieved a 14.56x speedup when scaling from 1 to 32 workers (128 GPUs).
  • Cost-Efficiency: Manoa was constructed for under $900,000, representing less than 45% of the cost of equivalent commercial solutions like the NVIDIA DGX-1 at the time of deployment.

5. Conclusion

The Manoa cluster and MiMatrix framework demonstrate that a co-design approach can overcome the limitations of traditional distributed training architectures. By combining a decentralized AllReduce algorithm with high-performance hardware, we achieved state-of-the-art results with superior price-to-performance ratios. Future work will focus on adaptive learning rate policies for large mini-batch training and scaling to larger datasets like ImageNet-22K.

Summary updated

Abstract

Large scale deep Convolution Neural Networks (CNNs) increasingly demands the computing power. It is key for researchers to own a great powerful computing platform to leverage deep learning (DL) advancing.On the other hand, as the commonly-used accelerator, the commodity GPUs cards of new generations are more and more expensive. Consequently, it is of importance to design an affordable distributed heterogeneous system that provides powerful computational capacity and develop a well-suited software that efficiently utilizes its computational capacity. In this paper, we present our co-design distributed system including a peta-scale GPU cluster, called "Manoa". Based on properties and topology of Manoa, we first propose job server framework and implement it, named "MiMatrix". The central node of MiMatrix, referred to as the job server, undertakes all of controlling, scheduling and monitoring, and I/O tasks without weight data transfer for AllReduce processing in each iteration. Therefore, MiMatrix intrinsically solves the bandwidth bottleneck of central node in parameter server framework that is widely used in distributed DL tasks. Meanwhile, we also propose a new AllReduce algorithm, GPUDirect RDMA-Aware AllReduce~(GDRAA), in which both computation and handshake message are O(1) and the number of synchronization is two in each iteration that is a theoretical minimum number. Owe to the dedicated co-design distributed system, MiMatrix efficiently makes use of the Manoa's computational capacity and bandwidth. We benchmark Manoa Resnet50 and Resenet101 on Imagenet-1K dataset. Some of results have demonstrated state-of-the-art.
BibTeX
@misc{chen2018novel,
  author = {Xin Chen and Hua Zhou and Yuxiang Gao and Yu Zhu},
  title = {{A Novel Co-design Peta-scale Heterogeneous Cluster for Deep Learning Training}},
  year = {2018},
  eprint = {1802.02326},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.02326},
}
Figure