Abstract
Improving the training and inference performance of graph neural networks (GNNs) is faced with a challenge uncommon in general neural networks: creating mini-batches requires a lot of computation and data movement due to the exponential growth of multi-hop graph neighborhoods along network layers. Such a unique challenge gives rise to a diverse set of system design choices. We argue in favor of performing mini-batch training with neighborhood sampling in a distributed multi-GPU environment, under which we identify major performance bottlenecks hitherto under-explored by developers: mini-batch preparation and transfer. We present a sequence of improvements to mitigate these bottlenecks, including a performance-engineered neighborhood sampler, a shared-memory parallelization strategy, and the pipelining of batch transfer with GPU computation. We also conduct an empirical analysis that supports the use of sampling for inference, showing that test accuracies are not materially compromised. Such an observation unifies training and inference, simplifying model implementation. We report comprehensive experimental results with several benchmark data sets and GNN architectures, including a demonstration that, for the ogbn-papers100M data set, our system SALIENT achieves a speedup of 3x over a standard PyTorch-Geometric implementation with a single GPU and a further 8x parallel speedup with 16 GPUs. Therein, training a 3-layer GraphSAGE model with sampling fanout (15, 10, 5) takes 2.0 seconds per epoch and inference with fanout (20, 20, 20) takes 2.4 seconds, attaining test accuracy 64.58%.
Introduction: The "Neighbor Explosion" Problem
Graph Neural Networks (GNNs) have emerged as a powerful paradigm for learning on non-Euclidean data, with applications spanning social network analysis, recommendation systems, and drug discovery. However, scaling GNNs to massive graphs presents a unique challenge: the "neighbor explosion" problem. Unlike traditional deep learning where mini-batch preparation is computationally inexpensive, GNNs require fetching multi-hop neighborhoods to compute the representation of a single node. As the number of layers increases, the required data grows exponentially.
In standard frameworks like PyTorch-Geometric (PyG), this leads to a significant imbalance. The CPU spends the majority of its time performing complex graph traversals and slicing feature vectors from main memory, while the GPU remains idle, waiting for data. Empirical analysis shows that CPU-side preparation and data transfer can account for 70-80% of total training time, effectively starving the GPU.
The SALIENT System
To address these bottlenecks, we introduce SALIENT (SAmpling, sLIcing, and data movemENT), a high-performance system designed to saturate GPU utilization through optimized data pipelines.
Fast Neighborhood Sampling
The core sampling logic was optimized by replacing standard C++ STL containers, such as std::unordered_set, with performance-engineered data structures. By utilizing "flat" hash tables and arrays optimized for cache locality, SALIENT achieves a 2.5x speedup in the sampling process itself.
Shared-Memory Parallelism
Standard Python multiprocessing often suffers from significant overhead due to data copying and the Global Interpreter Lock (GIL). SALIENT bypasses these limitations by using C++ multithreading to prepare batches within a shared memory space. This "zero-copy" approach allows seamless communication between CPU worker threads and the main training process, maximizing the throughput of prepared mini-batches.
Pipelining and Data Transfer
SALIENT implements a sophisticated pipelining strategy that overlaps three distinct stages: batch preparation on the CPU, host-to-device transfer via PCIe, and model computation on the GPU. This ensures that while the GPU is processing batch $N$, the data bus is transferring batch $N+1$, and the CPU workers are already sampling batch $N+2$.
By removing redundant assertions and optimizing round-trips in sparse libraries, SALIENT is able to utilize up to 99% of the peak PCIe bandwidth, ensuring the GPU is consistently supplied with data.
Unified Inference via Sampling
A common assumption in GNN research is that full-neighborhood aggregation is necessary during inference to maintain accuracy. However, SALIENT demonstrates that neighborhood sampling is also highly effective for inference. By using a modest fanout (e.g., 20 neighbors per layer), the system captures nearly all predictive power while drastically reducing memory requirements. This allows the same optimized pipeline to be used for both training and inference, simplifying implementation and enabling large-scale inference on a single GPU.
Experimental Results
The system was evaluated using the Open Graph Benchmark (OGB) datasets, including ogbn-arxiv, ogbn-products, and the massive ogbn-papers100M (111 million nodes, 1.6 billion edges).
- Performance: SALIENT achieved a 3x to 3.4x speedup over the baseline PyG implementation on a single GPU.
- Scalability: On
ogbn-papers100M, the system demonstrated excellent scaling, achieving an 8x parallel speedup when moving from 1 to 16 GPUs. - Efficiency: A 3-layer GraphSAGE model on the 111M node dataset was trained in just 2.0 seconds per epoch.
- Accuracy: The optimizations did not compromise model performance, maintaining a competitive test accuracy of 64.58% on the
ogbn-papers100Mbenchmark.
Conclusion
SALIENT demonstrates that the primary bottleneck in modern GNN training is a systems and data-movement challenge rather than a purely algorithmic one. By optimizing the CPU-to-GPU pipeline through fast sampling, shared-memory parallelism, and aggressive pipelining, SALIENT enables state-of-the-art performance on massive graph datasets, ensuring that high-performance hardware is fully utilized.
Summary updated