Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipelining

Tim Kaler, Nickolas Stathas, Anne Ouyang, Alexandros-Stavros Iliopoulos, Tao B. Schardl, Charles E. Leiserson, Jie Chen

TL;DR

This paper introduces SALIENT, a system that significantly accelerates the training and inference of Graph Neural Networks (GNNs) by optimizing the often-overlooked bottlenecks of mini-batch preparation and data transfer. SALIENT achieves this through a novel neighborhood sampler, shared-memory parallelization, and pipelined data transfers, leading to substantial speedups and enabling efficient GNN deployment on large graphs.

What Problem Does This Paper Address?

Training and deploying Graph Neural Networks (GNNs) on large graphs is computationally intensive. A unique challenge for GNNs, compared to standard neural networks, is the exponential growth of node neighborhoods with each network layer. This makes creating mini-batches for training and inference a complex and time-consuming process. Specifically, generating these mini-batches involves extensive computation and data movement to gather neighborhood information and slice relevant features. These "behind-the-scenes" operations, often referred to as batch preparation and transfer, have become major performance bottlenecks, frequently consuming more time than the core GNN computations on the GPU. This inefficiency hinders the scalability and practical application of GNNs to real-world large-scale graph data.

Methods

The SALIENT system tackles these bottlenecks with a multi-pronged approach:

  • Optimized Neighborhood Sampler: The authors developed a performance-engineered neighborhood sampler that explores a wide range of design choices to achieve high performance across different CPU architectures. This sampler is significantly faster than existing implementations.
  • Shared-Memory Parallelization: SALIENT utilizes shared-memory multithreading for batch preparation. This allows for zero-copy communication with the main training process and circumvents Python's Global Interpreter Lock (GIL), enabling more efficient slicing of feature and label tensors. Crucially, this allows slicing to occur directly into pinned memory accessible by the GPU, avoiding costly intermediate copies.
  • Pipelined Data Transfer: Data transfer from CPU to GPU is optimized by pipelining these operations with GPU computation. This means that while the GPU is busy processing one batch, the system is already preparing and transferring the next batch. This overlap significantly reduces GPU idle time.
  • Unified Training and Inference: The paper demonstrates that neighborhood sampling, commonly used for training, can also be effectively applied to inference without a significant loss in accuracy. This allows for code reuse and simplifies the overall GNN pipeline.

Datasets

The paper evaluates its approach on three standard benchmark datasets from the Open Graph Benchmark (OGB):

  • ogbn-arxiv
  • ogbn-products
  • ogbn-papers100M

These datasets vary in size, with ogbn-papers100M being one of the largest publicly available graph benchmarks at the time of publication.

Key Insights and Ideas

  • Batch preparation and transfer are the real bottlenecks: The paper's core insight is that for large-scale GNNs, the computational overhead of preparing mini-batches (sampling neighborhoods, slicing features) and transferring this data to the GPU often dominates the actual GNN computation time. This is a critical observation that shifts focus from optimizing the GNN model itself to optimizing the data pipeline.
  • Neighborhood sampling is viable for inference: Contrary to intuition, the authors show that using neighborhood sampling during inference does not significantly degrade prediction accuracy. This is a crucial finding that simplifies deployment by allowing the same sampling strategy used for training to be used for inference, reducing memory footprint and code complexity.
  • Shared memory is key for efficient data movement: By leveraging shared-memory multithreading and directly writing sliced data into pinned memory, SALIENT avoids expensive inter-process communication and memory copies, which are common in standard PyTorch implementations.
  • Pipelining is essential for high GPU utilization: Overlapping data transfers with GPU computations is critical to keep the expensive GPU hardware busy, minimizing idle time and maximizing throughput.

Key Results

  • Significant Speedups: SALIENT achieves a 3x speedup over a standard PyTorch-Geometric (PyG) implementation on a single GPU for the ogbn-papers100M dataset. With 16 GPUs, this speedup further increases to 8x.
  • Near-Optimal GPU Utilization: The optimizations in SALIENT effectively eliminate GPU idle time, making the training process nearly as fast as the core GPU computation itself.
  • Efficient Inference: For inference, SALIENT with sampling achieves comparable accuracy to full-neighborhood inference but with a drastically reduced memory footprint. For example, on ogbn-papers100M, training a 3-layer GraphSAGE model with sampling fanout (15, 10, 5) takes 2.0 seconds per epoch, and inference with fanout (20, 20, 20) takes 2.4 seconds, achieving a test accuracy of 64.58%.
  • Broad Applicability: The system is demonstrated to work with various GNN architectures (GraphSAGE, GAT, GIN) and scales well across multiple GPUs.

Limitations and Future Work

  • Hardware Dependency: While SALIENT significantly improves efficiency, the ultimate performance is still bound by hardware limitations, particularly CPU core count and DRAM bandwidth for batch preparation, and GPU memory bandwidth for data transfer. For extremely large graphs or feature sizes, further techniques like GPU-based slicing or caching might be necessary.
  • Distributed Graph Partitioning: The authors suggest that future work could explore distributing the graph and node data in a distributed computing environment to handle even larger graphs. This would involve re-investigating sampling approaches in a distributed setting to minimize communication overhead.
  • Feature Dimension Partitioning: For models with very long feature vectors, partitioning along the feature dimension is identified as another promising avenue for future work.

Summary updated

Abstract

Improving the training and inference performance of graph neural networks (GNNs) is faced with a challenge uncommon in general neural networks: creating mini-batches requires a lot of computation and data movement due to the exponential growth of multi-hop graph neighborhoods along network layers. Such a unique challenge gives rise to a diverse set of system design choices. We argue in favor of performing mini-batch training with neighborhood sampling in a distributed multi-GPU environment, under which we identify major performance bottlenecks hitherto under-explored by developers: mini-batch preparation and transfer. We present a sequence of improvements to mitigate these bottlenecks, including a performance-engineered neighborhood sampler, a shared-memory parallelization strategy, and the pipelining of batch transfer with GPU computation. We also conduct an empirical analysis that supports the use of sampling for inference, showing that test accuracies are not materially compromised. Such an observation unifies training and inference, simplifying model implementation. We report comprehensive experimental results with several benchmark data sets and GNN architectures, including a demonstration that, for the ogbn-papers100M data set, our system SALIENT achieves a speedup of 3x over a standard PyTorch-Geometric implementation with a single GPU and a further 8x parallel speedup with 16 GPUs. Therein, training a 3-layer GraphSAGE model with sampling fanout (15, 10, 5) takes 2.0 seconds per epoch and inference with fanout (20, 20, 20) takes 2.4 seconds, attaining test accuracy 64.58%.
BibTeX
@inproceedings{kaler2022accelerating,
  author = {Tim Kaler and Nickolas Stathas and Anne Ouyang and Alexandros-Stavros Iliopoulos and Tao B. Schardl and Charles E. Leiserson and Jie Chen},
  title = {{Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipelining}},
  booktitle = {MLSys},
  year = {2022},
  url = {https://openreview.net/forum?id=YJhTDWRiBs},
}

Loading related content...

Figure