Accelerating GNN Training on CPU+Multi-FPGA Heterogeneous Platform

Yi-Chien Lin, Bingyi Zhang, Viktor K. Prasanna

TL;DR

This paper introduces HitGNN, a framework that simplifies and accelerates Graph Neural Network (GNN) training on heterogeneous CPU+Multi-FPGA platforms. By automating hardware mapping and providing high-level APIs, HitGNN enables users to achieve significant speedups and bandwidth efficiency compared to multi-GPU systems, while using less power and memory.

What Problem Does This Paper Address?

Training Graph Neural Networks (GNNs) on large real-world graphs is computationally intensive and time-consuming. While FPGAs offer potential for acceleration due to their customizable hardware, developing GNN training accelerators for multi-FPGA systems is challenging. This is primarily due to the need for specialized hardware expertise, the complexity of managing multiple FPGAs, and the difficulty in balancing workloads and minimizing communication overhead. Existing solutions often require substantial development effort and lack generality.

Methods

HitGNN tackles these challenges by providing a comprehensive framework with several key components:

  • High-Level APIs: Users can specify GNN training algorithms and models with minimal code, abstracting away hardware implementation details.
  • Software Generator: This component generates a host program responsible for mini-batch sampling, managing CPU-FPGA communication, and balancing workloads across FPGAs.
  • Accelerator Generator: This module creates optimized GNN kernels for FPGAs, leveraging a pre-built kernel library and a Design Space Exploration (DSE) engine.
  • Design Space Exploration (DSE) Engine: The DSE engine automatically determines optimal hardware configurations for FPGAs by exploring various parameters (e.g., number of processing elements) based on platform metadata and performance models, aiming to maximize training throughput.
  • Optimizations:
    • Workload Balancing: A two-stage task scheduler is introduced to distribute mini-batches evenly across FPGAs, mitigating imbalance caused by graph partitioning.
  • Data Communication: To reduce slow FPGA-to-FPGA communication, data is fetched directly from the CPU's main memory when not found in local FPGA memory.
  • Optimized Kernel Library: Pre-built, parameterized hardware kernels for popular GNN models (like GraphSAGE and GCN) are provided to reduce development effort.

Datasets

The paper evaluates HitGNN on four widely-used datasets:

  • Reddit (RD)
  • Yelp (YP)
  • Amazon (AM)
  • ogbn-products (PR)

These datasets vary in size and number of edges, providing a comprehensive testbed for GNN training performance.

Key Insights and Ideas

  • Abstraction is Key: HitGNN's success hinges on its ability to abstract away hardware complexity, making GNN acceleration on multi-FPGA systems accessible to a broader audience.
  • DSE for Optimization: The automated Design Space Exploration is crucial for finding the optimal balance of hardware resources (e.g., scatter-gather PEs vs. update PEs) to maximize throughput, which isn't always intuitive.
  • CPU Memory as a Bottleneck and Enabler: While CPU memory bandwidth can limit scalability with many FPGAs, it also serves as a central hub for efficient data fetching, avoiding slower direct FPGA-to-FPGA communication.
  • Workload Balancing is Critical: The proposed two-stage scheduler significantly improves performance by ensuring FPGAs are kept busy, even when graph partitions have uneven numbers of mini-batches.

Key Results

  • Speedup: HitGNN achieves significant speedups compared to state-of-the-art multi-GPU baselines, with up to 2.34x for specific algorithms (P³).
  • Bandwidth Efficiency: The framework demonstrates superior bandwidth efficiency, achieving up to 14.9x higher throughput per GB/s of memory bandwidth compared to multi-GPU systems.
  • Resource Efficiency: HitGNN uses substantially less compute power and memory bandwidth than GPUs for comparable performance.
  • Scalability: The framework shows good scalability, with throughput increasing almost linearly up to 16 FPGAs.
  • Optimization Impact: An ablation study shows that workload balancing (WB) and data communication (DC) optimizations combined can deliver up to 66% throughput improvement.

Limitations and Future Work

  • Scalability Limit: The scalability of HitGNN is ultimately limited by the CPU's memory bandwidth, which can become saturated when connecting a large number of FPGAs.
  • Future Work: The authors suggest improving scalability by exploring data prefetching techniques and utilizing high-speed FPGA interconnection networks (e.g., SmartNICs) to offload the CPU memory bandwidth. They also aim to further enhance the DSE engine and expand the kernel library.

Summary updated

Abstract

Training Graph Neural Networks (GNNs) has become time consuming as the graphs grow larger. Thus, many works have been proposed to accelerate GNN training on multi-GPU platforms. Though GPUs feature high computation power, training GNNs on GPU suffers from low resource utilization. We propose to accelerate GNN training on a CPU+Multi-FPGA heterogeneous platform. By utilizing the customizable hardware resources on the FPGAs, we instantiate multiple hardware kernels with optimized data access pattern and memory organization. The optimized hardware kernels can efficiently access graph-structured data and thus achieve high training performance. However, training GNN with multiple FPGAs also leads to high FPGA-to-FPGA communication overhead and workload imbalance. We develop optimized graph partitioning techniques to minimize FPGA-to-FPGA data communication, and develop a task scheduler to balance the workload among the FPGAs. Compared with the state-of-the-art GNN training implementation on a multi-GPU platform, our work achieves up to 24.7\(\times \) bandwidth efficiency; this superior efficiency enables our work to achieve up to 3.88\(\times \) speedup and 7.18\(\times \) energy efficiency using much less compute power and memory bandwidth than GPUs.
BibTeX
@inproceedings{lin2022accelerating,
  author = {Yi-Chien Lin and Bingyi Zhang and Viktor K. Prasanna},
  title = {{Accelerating GNN Training on CPU+Multi-FPGA Heterogeneous Platform}},
  booktitle = {CARLA},
  year = {2022},
  doi = {10.1007/978-3-031-23821-5\_2},
}
Figure