Low-latency Mini-batch GNN Inference on CPU-FPGA Heterogeneous Platform

Bingyi Zhang, Hanqing Zeng, Viktor K. Prasanna

Abstract

Graph Neural Networks (GNNs) have emerged as powerful tools for graph-structured data, yet their deployment in real-time applications is hindered by high inference latency. This paper introduces a novel CPU-FPGA heterogeneous system designed to optimize mini-batch GNN inference. By integrating a decoupled GNN architecture with a flexible hardware accelerator featuring an Adaptive Computation Kernel (ACK), the system effectively addresses the "neighborhood explosion" and hardware load imbalance problems. Experimental results demonstrate speedups of up to 50x over CPUs and 21x over high-end GPUs, establishing a new state-of-the-art for low-latency GNN acceleration.

1. Introduction

Graph Neural Networks (GNNs) are increasingly critical for tasks such as recommendation systems, fraud detection, and social network analysis. However, deploying these models in production environments requires strict adherence to low-latency constraints. Traditional hardware architectures, including CPUs and GPUs, often struggle with the irregular memory access patterns and the computational diversity inherent in GNNs. This paper presents a co-designed hardware-software solution that leverages the strengths of both CPUs and FPGAs to achieve high-throughput, low-latency inference.

2. Challenges in GNN Acceleration

Two primary bottlenecks impede the performance of GNN inference:

2.1 Neighborhood Explosion

Standard GNNs utilize a recursive aggregation scheme where a node's representation is updated based on its $L$-hop neighbors. In many real-world graphs, this "receptive field" grows exponentially with the number of layers. This phenomenon, known as neighborhood explosion, results in massive data movement and computational overhead that quickly exceeds the memory bandwidth and capacity of standard hardware.

2.2 Hardware Load Imbalance

GNN layers typically consist of two distinct phases: Aggregation (sparse operations involving graph structure) and Transformation (dense operations involving feature matrix multiplications). Most existing hardware accelerators employ separate, dedicated modules for these tasks. Due to the irregular nature of graph data, these modules often experience significant load imbalance, where one module remains idle while the other is fully utilized, leading to suboptimal hardware efficiency.

3. Proposed Methodology

The authors propose a multi-level approach to overcome these challenges, spanning algorithmic decoupling and hardware innovation.

3.1 Model Depth-Receptive Field Decoupling

To mitigate neighborhood explosion, the system adopts "Decoupled GNNs." Instead of recursive neighbor searching, the system identifies a fixed set of "important neighbors" using Personalized PageRank (PPR). This creates a static, manageable subgraph for each target node. By decoupling the model depth from the receptive field, the computational complexity becomes linear relative to the number of layers, significantly reducing the data movement burden.

3.2 Adaptive Computation Kernel (ACK)

The core of the FPGA accelerator is the Adaptive Computation Kernel (ACK). Unlike rigid hybrid designs, the ACK is a unified processing element capable of reconfiguring its internal data paths:

  • Systolic Mode: Optimized for high-speed dense matrix multiplications during the transformation phase.
  • Scatter-Gather Mode: Optimized for sparse graph aggregations.

This flexibility ensures that hardware resources are fully utilized throughout the entire GNN layer execution, eliminating the idle time common in previous architectures.

3.3 Heterogeneous Task Scheduling

The system employs a CPU-FPGA pipeline where the CPU handles Important Neighbor Identification (INI) and subgraph construction, while the FPGA performs the heavy-duty inference. To maximize throughput, the authors implement double and triple buffering techniques to overlap data transfer with computation, effectively "hiding" communication latency.

4. Experimental Evaluation

The system was implemented on a Xilinx Alveo U250 board and evaluated against an AMD Ryzen 3990x CPU and an Nvidia RTX 3090 GPU.

4.1 Datasets and Models

The evaluation utilized three large-scale datasets:

  • Flickr (FL): ~90k vertices, ~900k edges.
  • Reddit (RE): ~233k vertices, ~116M edges.
  • ogbn-arxiv (OA): ~169k vertices, ~1.1M edges.

The system was tested across three popular GNN variants: GCN, GraphSAGE, and GAT.

5. Key Results and Insights

The proposed heterogeneous platform demonstrates significant performance gains:

  • Execution Speed: The system achieved 21.4x–50.8x speedup over CPU-only implementations and 2.9x–21.6x speedup over CPU-GPU configurations.
  • Comparison with State-of-the-Art: The design outperformed GraphACT, a leading FPGA-based GNN accelerator, by a factor of 4.7x.
  • Hardware Efficiency: The ACK design significantly improved hardware utilization. While previous hybrid designs suffered from up to 70% idle time, the ACK maintains high activity across both sparse and dense phases.
  • Scalability: The system maintains millisecond-level latency even for deep 16-layer models, making it suitable for real-time industrial applications.

6. Conclusion and Future Work

This work demonstrates that the combination of algorithmic decoupling and adaptive hardware kernels can overcome the fundamental bottlenecks of GNN inference. By transforming a memory-bound problem into a compute-bound one and ensuring high hardware utilization, the proposed system provides a viable path for deploying deep GNNs in latency-sensitive environments.

Future research directions include addressing potential CPU bottlenecks in neighbor identification for extremely large batches and extending the architecture to support the complexities of GNN training and backpropagation.

Summary updated

Abstract

Mini-batch inference of Graph Neural Networks (GNNs) is a key problem in many real-world applications. In this paper, we develop a computationally efficient mapping of GNNs onto CPU-FPGA heterogeneous platforms to achieve low-latency mini-batch inference. While the lightweight preprocessing algorithm of GNNs can be efficiently mapped onto the CPU platform, on the FPGA platform, we design a novel GNN hardware accelerator with an adaptive datapath denoted as Adaptive Computation Kernel (ACK) that can execute various computation kernels of GNNs with low-latency: (1) for dense computation kernels expressed as matrix multiplication, ACK works as a systolic array with fully localized connections, (2) for sparse computation kernels, ACK follows the scatter-gather paradigm and works as multiple parallel pipelines to support the irregular connectivity of graphs. The proposed task scheduling hides the CPU-FPGA data communication overhead to reduce the inference latency. We develop a fast design space exploration algorithm to generate a single accelerator for multiple target GNN models. We implement our accelerator on a state-of-the-art CPU-FPGA platform and evaluate the performance using three representative models (GCN, GraphSAGE, GAT). Results show that our CPU-FPGA implementation achieves 21.4−50.8×, 2.9 − 21.6×, 4.7× latency reduction compared with state-of-the-art implementations on CPU-only, CPU-GPU and CPU-FPGA platforms.
BibTeX
@inproceedings{zhang2022low,
  author = {Bingyi Zhang and Hanqing Zeng and Viktor K. Prasanna},
  title = {{Low-latency Mini-batch GNN Inference on CPU-FPGA Heterogeneous Platform}},
  booktitle = {HIPC},
  year = {2022},
  url = {https://openreview.net/forum?id=ZDLktJ0otl},
}
Figure