GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs on Large Clusters

Jaeyong Song, Hongsun Jang, Jaewon Jung, Youngsok Kim, Jinho Lee

Abstract

Graph neural networks (GNNs) are one of the most rapidly growing fields within deep learning. As dataset and model sizes grow, it becomes nearly impossible to keep the whole network on GPU memory. Distributed training is a popular approach to address this, but existing methods suffer from poor scalability due to slow external server communications. We propose GraNNDis, an efficient distributed GNN training framework. GraNNDis introduces three new techniques: shared preloading to reduce external communication, expansion-aware sampling to prune inter-server dependencies, and cooperative batching to reduce memory redundancy. Our experiments show that GraNNDis provides superior speedup over state-of-the-art frameworks on large-scale clusters.

Introduction

Training GNNs on massive graphs requires distributing the workload across multiple servers. However, existing frameworks face a "bandwidth wall." In a typical cluster, the connection between GPUs inside a single server (e.g., NVLink) is orders of magnitude faster than the connection between different servers (e.g., Ethernet or Infiniband).

Current distributed GNN approaches generally fall into two camps:

  1. Full-batch training: These methods suffer from an inter-server communication bottleneck because they frequently move massive amounts of data across slow network links.
  2. Mini-batch training: While they reduce communication, they suffer from "neighbor explosion," where dependencies grow exponentially with layers, leading to massive memory redundancy because multiple GPUs often store and compute the same vertex information independently.

GraNNDis bridges this gap by exploiting the massive bandwidth difference between intra-server and inter-server communications, treating the cluster as a hierarchical entity rather than a flat collection of GPUs.

Methodology: Exploiting Hierarchical Bandwidth

GraNNDis introduces a unified software architecture that optimizes data movement based on the physical location of resources.

GNN vertex dependency and hierarchical GPU cluster architecture.

The framework's core components include:

  • Flexible Preloading: Instead of fetching data layer-by-layer, GraNNDis preloads several hops of vertex dependencies into the aggregate memory of a server. This allows GPUs to fetch data from their local "neighbors" via fast intra-server links, bypassing the slow network.
  • Cooperative Batching: Rather than each GPU processing its own isolated mini-batch, GraNNDis aggregates GPUs within a server to work on a single, larger "co-batch." This significantly reduces redundant data storage and computation.
  • Expansion-aware Sampling: Traditional sampling thins out the entire graph. GraNNDis only samples edges that cross server boundaries. Since intra-server communication is "cheap," it keeps those dependencies intact to preserve accuracy, while aggressively pruning "expensive" inter-server dependencies to boost speed.
  • One-Hop Graph Masking: To implement these features efficiently, the authors developed a unified software architecture. It uses a static one-hop graph structure but applies dynamic "masks" to simulate different sampling or batching strategies, avoiding the high overhead of constant graph reshaping.

Experimental Evaluation

The framework was evaluated on six diverse datasets, ranging from standard benchmarks like ogbn-arxiv and ogbn-products to hyper-scale graphs like ogbn-papers100M (111 million nodes and 1.6 billion edges).

Key Results

  • Throughput & Speedup: GraNNDis provided a 1.92x to 4.00x speedup over optimized full-batch baselines (like PipeGCN) on high-end clusters. On clusters with slower 1GbE connections, the benefits were even more dramatic, showing 29x to 44x speedups, proving that the framework can make distributed GNN training viable on commodity hardware.
  • Scalability for Deep GNNs: GraNNDis successfully trained a 112-layer GNN on the Reddit dataset—a task that caused other state-of-the-art frameworks (like SALIENT++) to crash due to out-of-memory (OOM) errors.
  • Accuracy: Despite the sampling, GraNNDis maintained accuracy comparable to full-batch training (e.g., 69.21% vs 69.03% on Arxiv).

Discussion and Future Work

The primary insight of GraNNDis is that the "bandwidth gap" is an opportunity. By recognizing that "local-remote" data is almost free compared to "distant-remote" data, the framework can selectively sparsify the graph to eliminate the primary bottleneck while keeping the high-fidelity local graph structure that drives model accuracy.

While preloading requires additional GPU memory budget, the "flexible" approach manages this replication effectively. Future work may involve extending GraNNDis to support more diverse sampling methods and further optimizing the "one-hop masking" architecture for next-generation hardware like DGX-H100 systems.

Summary updated

Abstract

Graph neural networks (GNNs) are one of the most rapidly growing fields within deep learning. According to the growth in the dataset and the model size used for GNNs, an important problem is that it becomes nearly impossible to keep the whole network on GPU memory. Among numerous attempts, distributed training is one popular approach to address the problem. However, due to the nature of GNNs, existing distributed approaches suffer from poor scalability, mainly due to the slow external server communications. In this paper, we propose GraNNDis, an efficient distributed GNN training framework for training GNNs on large graphs and deep layers. GraNNDis introduces three new techniques. First, shared preloading provides a training structure for a cluster of multi-GPU servers. We suggest server-wise preloading of essential vertex dependencies to reduce the low-bandwidth external server communications. Second, we present expansion-aware sampling. Because shared preloading alone has limitations because of the neighbor explosion, expansion-aware sampling reduces vertex dependencies that span across server boundaries. Third, we propose cooperative batching to create a unified framework for full-graph and minibatch training. It significantly reduces redundant memory usage in mini-batch training. From this, GraNNDis enables a reasonable trade-off between full-graph and mini-batch training through unification especially when the entire graph does not fit into the GPU memory. With experiments conducted on a multi-server/multi-GPU cluster, we show that GraNNDis provides superior speedup over the state-of-the-art distributed GNN training frameworks.
BibTeX
@inproceedings{song2024granndis,
  author = {Jaeyong Song and Hongsun Jang and Jaewon Jung and Youngsok Kim and Jinho Lee},
  title = {{GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs on Large Clusters}},
  booktitle = {PACT},
  year = {2024},
  url = {https://openreview.net/forum?id=Yup2D15wHo},
}
Figure