Introduction
While Convolutional Neural Networks (CNNs) and Transformers have long dominated the computer vision landscape, Vision Graph Neural Networks (ViGs) have emerged as a compelling alternative. By representing images as graphs of patches rather than rigid grids, ViGs can capture complex, non-local relationships with greater flexibility. However, this flexibility comes at a significant computational cost: the graph structure is not static. In a process known as Dynamic Image Graph Construction (DIGC), the network must rebuild the graph at every layer to account for evolving features.
Research from USC identifies DIGC as a massive computational bottleneck, often accounting for 50% to 95% of total inference latency. To address this, the researchers developed a specialized FPGA-based accelerator designed to handle the irregular memory patterns and high data movement inherent in graph construction.
The Graph Construction Bottleneck
DIGC involves calculating distances between all image patches, sorting these distances, and selecting the nearest neighbors to form edges. As image resolution increases, the number of nodes grows, but the potential edges increase quadratically. This "Resolution Wall" causes standard CPUs and GPUs to struggle due to their reliance on sequential execution and external memory for intermediate results. At high resolutions (e.g., 2048×2048), these platforms often suffer from severe performance degradation or "Out of Memory" (OOM) errors.
Proposed Architecture: A Streaming Pipeline
The proposed solution is a modular, deeply pipelined FPGA architecture implemented on a Xilinx Alveo U280. Unlike traditional hardware that offloads intermediate data to DDR memory, this design utilizes a fully streaming pipeline to keep data on-chip, drastically reducing external memory traffic. The architecture consists of four primary modules:
- Distance Computation Module (DCM): Utilizes a 2D mesh of Processing Elements (PEs) to calculate squared Euclidean distances between patches in parallel.
- Local Sorting Module (LSM): Executes high-throughput parallel merge-sorts on small data blocks stored in local on-chip buffers.
- Global Merging Module (GMM): Employs a distributed k-way min-heap to merge locally sorted lists into a globally sorted candidate list.
- Neighbor Selection Module (NSM): Implements "dilated" k-nearest neighbor selection to expand the network's receptive field without increasing the edge count.
By processing input features in uniform blocks and maintaining distance-index pairs on-chip, the design circumvents the memory bandwidth limitations that typically cripple GPUs during irregular workloads.
Experimental Results
The accelerator was evaluated against high-end hardware baselines, including a 128-core AMD EPYC CPU and an NVIDIA RTX A5000 GPU, using various ViG model sizes (Tiny to Base) and resolutions ranging from 256×256 to 2048×2048.
Performance and Scalability
The FPGA accelerator achieved a post-place-and-route clock frequency of 600 MHz, a remarkably high figure for HLS-based designs. Key performance metrics include:
- Speedup: The design delivered up to a 16.6× speedup over optimized CPUs and a 6.8× speedup over GPUs for the DIGC operation.
- End-to-End Impact: Integrating the accelerator into a full ViG model resulted in an overall inference acceleration of 2.1× to 4.6×.
- Robustness at Scale: While the GPU baselines failed at 2048×2048 resolution due to memory exhaustion, the FPGA’s partitioned memory approach allowed it to scale successfully, making it ideal for high-resolution industrial or medical imaging.
Conclusion and Future Work
This research demonstrates that hardware-algorithm co-design is essential for the practical deployment of Vision GNNs. By tailoring the hardware to the specific data flow of graph construction, the researchers eliminated the primary bottleneck of ViG models.
Future iterations of this work could expand the design to support alternative similarity metrics beyond Euclidean distance, such as cosine similarity. Additionally, there is potential to integrate the GNN message-passing layers onto the same FPGA, creating a fully self-contained System-on-Chip (SoC) for graph-based vision tasks.
Summary updated