SuperGCN: General and Scalable Framework for GCN Training on CPU-powered Supercomputers

Chen Zhuang, Peng Chen, Xin Liu, Rio Yokota, Nikoli Dryden, Toshio Endo, Satoshi Matsuoka, Mohamed WahibarXiv 2024 · arXiv · OpenReview

Abstract

Graph Convolutional Networks (GCNs) are widely used in various domains. However, training distributed full-batch GCNs on large-scale graphs poses challenges due to inefficient memory access patterns and high communication overhead. This paper presents general and efficient aggregation operators designed for irregular memory access patterns. Additionally, we propose a pre-post-aggregation approach and a quantization with label propagation method to reduce communication costs. Combining these techniques, we develop an efficient and scalable distributed GCN training framework, SuperGCN, for CPU-powered supercomputers. Experimental results on multiple large graph datasets show that our method achieves a speedup of up to 6× compared with the SoTA implementations, and scales to 1000s of HPC-grade CPUs, without sacrificing model convergence and accuracy. Our framework achieves performance on CPU-powered supercomputers comparable to that of GPU-powered supercomputers, with a fraction of the cost and power budget.

Introduction: The Challenge of Large-Scale GCNs

Training Graph Convolutional Networks (GCNs) on massive graphs containing billions of edges is a significant challenge for traditional computing architectures. Unlike standard deep learning tasks where data is neatly structured in tensors, graph data is inherently irregular. When distributing a giant graph across a supercomputer, two major bottlenecks emerge:

  1. Irregular Memory Access: The random connectivity of graphs leads to poor cache utilization on CPUs, as memory access patterns do not follow linear or predictable paths.
  2. Communication Overhead: To compute a single layer, processors must constantly exchange "boundary node" features across the network. As the graph size grows, the volume of this inter-processor communication often becomes the primary limiting factor for scalability.
Min-cut partitioning of a graph into subgraphs $S_0$ and $S_1$, highlighting the boundary nodes that necessitate inter-processor communication.

While many existing solutions rely on mini-batching—sampling the graph to reduce computation—this can destroy structural information and degrade model accuracy. SuperGCN aims to make full-batch training viable and fast on massive CPU-based clusters like Fugaku and ABCI.

The SuperGCN Framework

SuperGCN introduces three primary innovations to overcome the limitations of distributed GCN training: hardware-optimized aggregation, a hierarchical aggregation scheme, and communication-aware quantization.

The end-to-end workflow of SuperGCN, illustrating graph splitting, pre-post aggregation, masked label propagation, and quantized communication.

Hardware-Optimized Aggregation

The framework utilizes custom CPU-level operators designed for both x86 and Arm architectures. These operators employ clustering, sorting, and loop reordering techniques to ensure that data remains in the high-speed cache for as long as possible, significantly improving the efficiency of irregular memory access during the aggregation phase.

Hierarchical Aggregation and the MVC Strategy

One of the paper's most novel contributions is treating the communication bottleneck as a graph theory problem. By identifying the Minimum Vertex Cover (MVC) of the "cut edges" between processors, the framework mathematically determines the most efficient communication path. This hybrid approach allows the system to decide whether it is more efficient to aggregate data before sending it (pre-aggregation) or after it arrives (post-aggregation), minimizing the total number of bytes moved across the network.

Communication-Aware Quantization and Recovery

To further reduce network traffic, SuperGCN compresses feature vectors down to just 2 bits (Int2). To mitigate the accuracy drop typically associated with such aggressive compression, the authors integrate Masked Label Propagation (MLP). This technique leverages known node labels during training to help the model "recover" lost precision, acting as a regularizer that maintains predictive power despite the low-precision communication.

Experimental Evaluation

The framework was evaluated on a wide variety of benchmarks, including standard datasets like Reddit and Ogbn-products, as well as extreme-scale datasets like IGB260M (269 million nodes, 3.9 billion edges) and Ogb-lsc-mag240M.

Key Results

  • Speedup: SuperGCN achieved up to a 6× speedup over DistGNN (the previous state-of-the-art) on Intel-based systems.
  • Scalability: The system successfully scaled to 8,192 MPI ranks on the Fugaku supercomputer, demonstrating its ability to handle massive parallelization.
  • Accuracy: Despite the use of Int2 quantization, the framework maintained or slightly exceeded the accuracy of full-precision (FP32) baselines, proving that precision can be traded for speed without sacrificing performance.
  • CPU vs. GPU: The results indicate that for giant graphs, the massive memory capacity and sophisticated cache hierarchies of HPC-grade CPUs (like the Arm A64FX) can outperform GPUs, which frequently encounter out-of-memory (OOM) errors on these datasets.

Conclusion

SuperGCN demonstrates that CPU-powered supercomputers can be highly effective for large-scale GCN training when paired with hardware-aware optimizations and clever communication strategies. By solving the communication bottleneck through graph theory and utilizing masked label propagation to recover quantization losses, SuperGCN provides a scalable, cost-effective alternative to GPU-heavy distributed frameworks for the world's largest graph datasets.

Summary updated

Abstract

Graph Convolutional Networks (GCNs) are widely used in various domains. However, training distributed full-batch GCNs on large-scale graphs poses challenges due to inefficient memory access patterns and high communication overhead. This paper presents general and efficient aggregation operators designed for irregular memory access patterns. Additionally, we propose a pre-post-aggregation approach and a quantization with label propagation method to reduce communication costs. Combining these techniques, we develop an efficient and scalable distributed GCN training framework, \emph{SuperGCN}, for CPU-powered supercomputers. Experimental results on multiple large graph datasets show that our method achieves a speedup of up to 6$\times$ compared with the SoTA implementations, and scales to 1000s of HPC-grade CPUs, without sacrificing model convergence and accuracy. Our framework achieves performance on CPU-powered supercomputers comparable to that of GPU-powered supercomputers, with a fraction of the cost and power budget.
BibTeX
@misc{zhuang2024supergcn,
  author = {Chen Zhuang and Peng Chen and Xin Liu and Rio Yokota and Nikoli Dryden and Toshio Endo and Satoshi Matsuoka and Mohamed Wahib},
  title = {{SuperGCN: General and Scalable Framework for GCN Training on CPU-powered Supercomputers}},
  year = {2024},
  eprint = {2411.16025},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.16025},
}
Figure