Abstract
Graph neural networks (GNNs) are one of the most rapidly growing fields within deep learning. As dataset and model sizes grow, it becomes nearly impossible to keep the whole network on GPU memory. Distributed training is a popular approach to address this, but existing methods suffer from poor scalability due to slow external server communications. We propose GraNNDis, an efficient distributed GNN training framework. GraNNDis introduces three new techniques: shared preloading to reduce external communication, expansion-aware sampling to prune inter-server dependencies, and cooperative batching to reduce memory redundancy. Our experiments show that GraNNDis provides superior speedup over state-of-the-art frameworks on large-scale clusters.
Introduction
Training GNNs on massive graphs requires distributing the workload across multiple servers. However, existing frameworks face a "bandwidth wall." In a typical cluster, the connection between GPUs inside a single server (e.g., NVLink) is orders of magnitude faster than the connection between different servers (e.g., Ethernet or Infiniband).
Current distributed GNN approaches generally fall into two camps:
- Full-batch training: These methods suffer from an inter-server communication bottleneck because they frequently move massive amounts of data across slow network links.
- Mini-batch training: While they reduce communication, they suffer from "neighbor explosion," where dependencies grow exponentially with layers, leading to massive memory redundancy because multiple GPUs often store and compute the same vertex information independently.
GraNNDis bridges this gap by exploiting the massive bandwidth difference between intra-server and inter-server communications, treating the cluster as a hierarchical entity rather than a flat collection of GPUs.
Methodology: Exploiting Hierarchical Bandwidth
GraNNDis introduces a unified software architecture that optimizes data movement based on the physical location of resources.
The framework's core components include:
- Flexible Preloading: Instead of fetching data layer-by-layer, GraNNDis preloads several hops of vertex dependencies into the aggregate memory of a server. This allows GPUs to fetch data from their local "neighbors" via fast intra-server links, bypassing the slow network.
- Cooperative Batching: Rather than each GPU processing its own isolated mini-batch, GraNNDis aggregates GPUs within a server to work on a single, larger "co-batch." This significantly reduces redundant data storage and computation.
- Expansion-aware Sampling: Traditional sampling thins out the entire graph. GraNNDis only samples edges that cross server boundaries. Since intra-server communication is "cheap," it keeps those dependencies intact to preserve accuracy, while aggressively pruning "expensive" inter-server dependencies to boost speed.
- One-Hop Graph Masking: To implement these features efficiently, the authors developed a unified software architecture. It uses a static one-hop graph structure but applies dynamic "masks" to simulate different sampling or batching strategies, avoiding the high overhead of constant graph reshaping.
Experimental Evaluation
The framework was evaluated on six diverse datasets, ranging from standard benchmarks like ogbn-arxiv and ogbn-products to hyper-scale graphs like ogbn-papers100M (111 million nodes and 1.6 billion edges).
Key Results
- Throughput & Speedup: GraNNDis provided a 1.92x to 4.00x speedup over optimized full-batch baselines (like PipeGCN) on high-end clusters. On clusters with slower 1GbE connections, the benefits were even more dramatic, showing 29x to 44x speedups, proving that the framework can make distributed GNN training viable on commodity hardware.
- Scalability for Deep GNNs: GraNNDis successfully trained a 112-layer GNN on the Reddit dataset—a task that caused other state-of-the-art frameworks (like SALIENT++) to crash due to out-of-memory (OOM) errors.
- Accuracy: Despite the sampling, GraNNDis maintained accuracy comparable to full-batch training (e.g., 69.21% vs 69.03% on Arxiv).
Discussion and Future Work
The primary insight of GraNNDis is that the "bandwidth gap" is an opportunity. By recognizing that "local-remote" data is almost free compared to "distant-remote" data, the framework can selectively sparsify the graph to eliminate the primary bottleneck while keeping the high-fidelity local graph structure that drives model accuracy.
While preloading requires additional GPU memory budget, the "flexible" approach manages this replication effectively. Future work may involve extending GraNNDis to support more diverse sampling methods and further optimizing the "one-hop masking" architecture for next-generation hardware like DGX-H100 systems.
Summary updated