Abstract
Graph Convolutional Networks (GCNs) are widely used in various domains. However, training distributed full-batch GCNs on large-scale graphs poses challenges due to inefficient memory access patterns and high communication overhead. This paper presents general and efficient aggregation operators designed for irregular memory access patterns. Additionally, we propose a pre-post-aggregation approach and a quantization with label propagation method to reduce communication costs. Combining these techniques, we develop an efficient and scalable distributed GCN training framework, SuperGCN, for CPU-powered supercomputers. Experimental results on multiple large graph datasets show that our method achieves a speedup of up to 6× compared with the SoTA implementations, and scales to 1000s of HPC-grade CPUs, without sacrificing model convergence and accuracy. Our framework achieves performance on CPU-powered supercomputers comparable to that of GPU-powered supercomputers, with a fraction of the cost and power budget.
Introduction: The Challenge of Large-Scale GCNs
Training Graph Convolutional Networks (GCNs) on massive graphs containing billions of edges is a significant challenge for traditional computing architectures. Unlike standard deep learning tasks where data is neatly structured in tensors, graph data is inherently irregular. When distributing a giant graph across a supercomputer, two major bottlenecks emerge:
- Irregular Memory Access: The random connectivity of graphs leads to poor cache utilization on CPUs, as memory access patterns do not follow linear or predictable paths.
- Communication Overhead: To compute a single layer, processors must constantly exchange "boundary node" features across the network. As the graph size grows, the volume of this inter-processor communication often becomes the primary limiting factor for scalability.
While many existing solutions rely on mini-batching—sampling the graph to reduce computation—this can destroy structural information and degrade model accuracy. SuperGCN aims to make full-batch training viable and fast on massive CPU-based clusters like Fugaku and ABCI.
The SuperGCN Framework
SuperGCN introduces three primary innovations to overcome the limitations of distributed GCN training: hardware-optimized aggregation, a hierarchical aggregation scheme, and communication-aware quantization.
Hardware-Optimized Aggregation
The framework utilizes custom CPU-level operators designed for both x86 and Arm architectures. These operators employ clustering, sorting, and loop reordering techniques to ensure that data remains in the high-speed cache for as long as possible, significantly improving the efficiency of irregular memory access during the aggregation phase.
Hierarchical Aggregation and the MVC Strategy
One of the paper's most novel contributions is treating the communication bottleneck as a graph theory problem. By identifying the Minimum Vertex Cover (MVC) of the "cut edges" between processors, the framework mathematically determines the most efficient communication path. This hybrid approach allows the system to decide whether it is more efficient to aggregate data before sending it (pre-aggregation) or after it arrives (post-aggregation), minimizing the total number of bytes moved across the network.
Communication-Aware Quantization and Recovery
To further reduce network traffic, SuperGCN compresses feature vectors down to just 2 bits (Int2). To mitigate the accuracy drop typically associated with such aggressive compression, the authors integrate Masked Label Propagation (MLP). This technique leverages known node labels during training to help the model "recover" lost precision, acting as a regularizer that maintains predictive power despite the low-precision communication.
Experimental Evaluation
The framework was evaluated on a wide variety of benchmarks, including standard datasets like Reddit and Ogbn-products, as well as extreme-scale datasets like IGB260M (269 million nodes, 3.9 billion edges) and Ogb-lsc-mag240M.
Key Results
- Speedup: SuperGCN achieved up to a 6× speedup over DistGNN (the previous state-of-the-art) on Intel-based systems.
- Scalability: The system successfully scaled to 8,192 MPI ranks on the Fugaku supercomputer, demonstrating its ability to handle massive parallelization.
- Accuracy: Despite the use of Int2 quantization, the framework maintained or slightly exceeded the accuracy of full-precision (FP32) baselines, proving that precision can be traded for speed without sacrificing performance.
- CPU vs. GPU: The results indicate that for giant graphs, the massive memory capacity and sophisticated cache hierarchies of HPC-grade CPUs (like the Arm A64FX) can outperform GPUs, which frequently encounter out-of-memory (OOM) errors on these datasets.
Conclusion
SuperGCN demonstrates that CPU-powered supercomputers can be highly effective for large-scale GCN training when paired with hardware-aware optimizations and clever communication strategies. By solving the communication bottleneck through graph theory and utilizing masked label propagation to recover quantization losses, SuperGCN provides a scalable, cost-effective alternative to GPU-heavy distributed frameworks for the world's largest graph datasets.
Summary updated