fpgaHART: A Toolflow for Throughput-Oriented Acceleration of 3D CNNs for HAR onto FPGAs

Petros Toupas, Christos-Savvas Bouganis, Dimitrios Tzovaras

Abstract

Human Action Recognition (HAR) relies heavily on 3D Convolutional Neural Networks (CNNs) to capture temporal dynamics, yet these models are computationally demanding for edge deployment. We present fpgaHART, an automated toolflow that maps complex 3D CNNs from high-level frameworks to FPGA hardware. By extending Synchronous Dataflow (SDF) modeling to accommodate modern branching architectures and employing intelligent model partitioning, fpgaHART achieves high throughput across diverse FPGA platforms. Our results demonstrate that fpgaHART can deliver competitive performance, such as 13.44 clips/s for X3D on a ZCU102, while maintaining the flexibility to support state-of-the-art non-sequential architectures.

Introduction

Human Action Recognition (HAR) is a cornerstone of modern computer vision applications, ranging from autonomous driving to intelligent surveillance. Unlike static image tasks, HAR requires 3D CNNs to process the temporal dimension of motion. However, the computational and memory intensity of 3D CNNs poses significant challenges for deployment on resource-constrained edge devices.

While FPGAs offer a low-power, flexible alternative to GPUs, they are notoriously difficult to program. Existing FPGA accelerators often suffer from model rigidity—targeting older sequential models like C3D—and fail to optimize for throughput, which is the primary metric for HAR where videos are processed as multiple overlapping clips.

The fpgaHART Toolflow

The authors introduce fpgaHART, a toolflow designed to automate the transition from PyTorch or ONNX models to synthesisable Vitis HLS code. The system is built on several key technical pillars:

  • Extended SDFG Modeling: Standard Synchronous Dataflow Graphs (SDFG) are extended to handle the branching behaviors characteristic of modern architectures like ResNet and X3D.
  • Automated Partitioning: To handle models larger than the available on-chip memory, fpgaHART automatically splits networks into partitions. These are loaded sequentially, maximizing data reuse and minimizing off-chip memory bottlenecks.
  • Design Space Exploration (DSE): A Simulated Annealing algorithm optimizes hardware configurations—such as parallelism levels and buffer sizes—to maximize throughput within specific FPGA resource constraints.
  • Hardware Templates: Layers are mapped to parameterized templates that exploit both coarse-grained (channel-level) and fine-grained (dot-product level) parallelism.

Evaluation and Datasets

The toolflow was evaluated using the Kinetics-400 and UCF-101 benchmarks. UCF-101 served as the primary dataset for evaluating model accuracy after conversion to 16-bit fixed-point arithmetic for FPGA deployment. The study covered five distinct 3D CNN architectures: C3D, Slowonly, R(2+1)D-18, R(2+1)D-34, and X3D.

Key Results and Performance

fpgaHART demonstrates broad compatibility, successfully mapping architectures across multiple FPGA platforms including the ZCU104, ZCU102, VC709, and VUS440.

Accuracy vs. Throughput (Clips/s) comparison of fpgaHART against other works.

On the state-of-the-art X3D model, fpgaHART achieved a throughput of 13.44 clips/s on a ZCU102. When compared to high-end GPUs like the Nvidia RTX 3090, the FPGA solution consumed significantly less power (26W vs. ~300W), making it a viable candidate for power-sensitive edge environments.

Throughput in GOPs/s for different 3D CNN models across various FPGA platforms.

Discussion: Insights and Innovations

A core breakthrough of fpgaHART is its ability to handle "branching" topologies. By mathematically incorporating branching into the SDFG topology matrix, the tool can optimize modern, efficient networks that previous linear-pipeline toolflows could not.

Furthermore, the tool prioritizes throughput over single-frame latency. Given that HAR evaluation typically involves averaging predictions across multiple clips per video, a streaming architecture that maximizes clips-per-second is more aligned with real-world requirements. The tool also embraces dynamic reconfiguration; by loading model stages sequentially, each stage can utilize more hardware resources for higher parallelism, ultimately boosting total throughput.

Limitations and Future Work

While fpgaHART excels in throughput and total power consumption, the authors noted that high-end GPUs still maintain a lead in "clips per watt" efficiency. Future work will focus on optimizing hardware templates for better energy efficiency and exploring lower arithmetic precision (e.g., 8-bit or 4-bit) to further enhance performance. Additionally, incorporating latency-driven optimizations could broaden the tool's applicability to real-time response scenarios.

Summary updated

BibTeX
@inproceedings{toupas2023fpgahart,
  author = {Petros Toupas and Christos-Savvas Bouganis and Dimitrios Tzovaras},
  title = {{fpgaHART: A Toolflow for Throughput-Oriented Acceleration of 3D CNNs for HAR onto FPGAs}},
  booktitle = {FPL},
  year = {2023},
  doi = {10.1109/fpl60245.2023.00020},
}
Figure