GraNNDis: Fast Distributed Graph Neural Network Training Framework for Multi-Server Clusters

Jaeyong Song, Hongsun Jang, Hunseong Lim, Jaewon Jung, Youngsok Kim, Jinho Lee

TL;DR

This paper introduces GraNNDis, a novel framework for accelerating distributed Graph Neural Network (GNN) training on multi-server clusters. By intelligently exploiting the differences in communication speeds between servers (inter-server) and within servers (intra-server), GraNNDis significantly reduces training time and memory usage compared to existing methods.

What Problem Does This Paper Address?

Graph Neural Networks (GNNs) are powerful tools for analyzing graph-structured data, finding applications in areas like social network analysis, drug discovery, and recommendation systems. As GNN models grow deeper and graphs become larger, distributed training becomes essential to handle the computational and memory demands. However, existing distributed GNN training frameworks struggle with multi-server clusters due to three main issues:

  1. Inter-server Communication Bottleneck: Multi-server clusters have a significant disparity in communication speeds. Fast intra-server links (like NVLink) are orders of magnitude faster than slower inter-server links (like Ethernet). Existing frameworks often treat all communication equally, leading to bottlenecks on the slower inter-server links.
  2. Redundant Memory Usage and Computation: Standard distributed training approaches can lead to duplicated computations and memory footprints across different servers, hindering scalability.
  3. Sampling Inefficiencies: While sampling is crucial for managing large graphs in mini-batch training, existing methods can introduce unnecessary errors when applied in a multi-server context, as they don't account for the intra-server bandwidth advantage.

Methods

GraNNDis tackles these challenges with a suite of novel techniques:

  • Flexible Preloading: This method intelligently preloads essential vertex dependencies onto each server, prioritizing the use of fast intra-server links to minimize reliance on slower inter-server communication. It dynamically adjusts preloading based on available server memory.
  • Cooperative Batching: This technique enables memory-efficient and less redundant mini-batch training by aggregating GPUs within a server. Instead of each GPU processing its own mini-batch independently, cooperative batching allows them to work together, creating larger, more efficient batches.
  • Expansion-aware Sampling: This cluster-aware sampling strategy focuses on sampling edges that are critical for system speedup. It intelligently targets only the "server boundary" vertices for sampling, recognizing that sampling intra-server dependencies offers little benefit due to their fast communication.
  • One-Hop Graph Masking: This is a core structural component that unifies the above methods. It provides an efficient data structure and software architecture to realize flexible preloading and cooperative batching in multi-server environments, supporting both full-batch and mini-batch training.

Datasets

The paper evaluates GraNNDis on several standard graph datasets:

  • ogbn-arxiv
  • ogbn-products
  • Reddit
  • IGB (S)
  • IGB (M)
  • ogbn-papers100M (for hyper-scale evaluation)

Key Insights and Ideas

  • Exploiting the Intra-/Inter-Server Bandwidth Gap is Crucial: The most significant insight is that the performance of distributed GNN training in multi-server clusters is heavily dictated by the difference in communication speeds. Ignoring this gap leads to suboptimal performance.
  • Server-Level Aggregation is Key for Mini-Batch Efficiency: By treating GPUs within a server as a cooperative unit, GraNNDis significantly reduces redundancy in mini-batch training, enabling larger batch sizes and better memory utilization.
  • Targeted Sampling for Multi-Server Clusters: Standard sampling methods are often too broad for multi-server setups. Expansion-aware sampling's focus on server boundaries is a more efficient approach that balances speedup with accuracy.
  • Unified Framework for Full- and Mini-Batch: The one-hop graph masking provides a flexible architecture that allows GraNNDis to seamlessly switch between or combine full-batch and mini-batch strategies, adapting to different computational needs and memory constraints.

Key Results

GraNNDis demonstrates substantial improvements over state-of-the-art distributed GNN training frameworks:

  • Significant Speedup: GraNNDis achieves considerable speedups across various datasets and GNN depths. For example, in full-batch training, flexible preloading (FLX) provides 1.11-2.96x speedup, and with expansion-aware sampling (FLX-EAS), speedups range from 1.85-3.56x.
  • Reduced Communication Overhead: The framework effectively minimizes inter-server communication, which often constitutes a large portion of training time in existing methods (e.g., 44.07-85.81% in FB_base).
  • Memory Efficiency: Cooperative batching significantly reduces redundant memory usage, enabling the training of deeper GNNs and larger graphs that would otherwise cause out-of-memory errors. For instance, on a hyper-scale dataset, GraNNDis was able to train models that baselines could not.
  • Comparable Accuracy: Despite the focus on speed, GraNNDis achieves comparable or even better accuracy than baseline methods, particularly with expansion-aware sampling.

Limitations and Future Work

  • Hardware Dependency: The performance gains are contingent on the specific characteristics of multi-server clusters, particularly the bandwidth difference between inter- and intra-server links.
  • Parameter Tuning: While expansion-aware sampling offers flexibility, optimal performance might require careful tuning of its hyperparameters (hop count, fanout).
  • Future Work: The authors suggest that their framework could be further enhanced by integrating with other advanced techniques like pipelining (though they note potential staleness issues) and by exploring its application on even larger-scale systems with faster interconnects. The open-sourced nature of GraNNDis also facilitates future research and development.

Summary updated

Abstract

Graph neural networks (GNNs) are one of the rapidly growing fields within deep learning. While many distributed GNN training frameworks have been proposed to increase the training throughput, they face three limitations when applied to multi-server clusters. 1) They suffer from an inter-server communication bottleneck because they do not consider the inter-/intra-server bandwidth gap, a representative characteristic of multi-server clusters. 2) Redundant memory usage and computation hinder the scalability of the distributed frameworks. 3) Sampling methods, de facto standard in mini-batch training, incur unnecessary errors in multi-server clusters.We found that these limitations can be addressed by exploiting the characteristics of multi-server clusters. Here, we propose GraNNDis, a fast distributed GNN training framework for multi-server clusters. Firstly, we present Flexible Preloading, which preloads the essential vertex dependencies server-wise to reduce the low-bandwidth inter-server communications. Secondly, we introduce Cooperative Batching, which enables memory-efficient, less redundant mini-batch training by utilizing high-bandwidth intra-server communications. Thirdly, we propose Expansion-aware Sampling, a cluster-aware sampling method, which samples the edges that affect the system speedup. As sampling the intra-server dependencies does not contribute much to the speedup as they are communicated through fast intra-server links, it only targets a server boundary to be sampled. Lastly, we introduce One-Hop Graph Masking, a computation and communication structure to realize the above methods in multi-server environments. We evaluated GraNNDis on multi-server clusters, and it provided significant speedup over the state-of-the-art distributed GNN training frameworks. GraNNDis is open-sourced at https://github.com/AIS-SNU/GraNNDis_Artifact to facilitate its use.
BibTeX
@inproceedings{song2024granndis,
  author = {Jaeyong Song and Hongsun Jang and Hunseong Lim and Jaewon Jung and Youngsok Kim and Jinho Lee},
  title = {{GraNNDis: Fast Distributed Graph Neural Network Training Framework for Multi-Server Clusters}},
  booktitle = {PACT},
  year = {2024},
  url = {https://openreview.net/forum?id=Yup2D15wHo},
}
Figure