SATAY: A Streaming Architecture Toolflow for Accelerating YOLO Models on FPGA Devices

Alexander Montgomerie-Corcoran, Petros Toupas, Zhewen Yu, Christos-Savvas Bouganis

Abstract

SATAY is an automated toolflow designed to map the entire YOLO family (v3 to v8) onto FPGA devices using a high-performance, deeply pipelined streaming architecture. By maintaining model weights on-chip and selectively offloading large skip-connection buffers to external memory, SATAY achieves ultra-low latency and superior performance density compared to existing FPGA and edge-GPU solutions. This work presents the first FPGA implementation of YOLOv8, demonstrating significant improvements in the trade-off between accuracy and hardware efficiency.

Introduction

Object detection is a critical component of autonomous systems, with the YOLO (You Only Look Once) family of models being the industry standard. However, the increasing complexity of modern YOLO versions (v5-v8) introduces sophisticated layers, such as SiLU activations and CSP blocks, along with long skip connections that pose significant challenges for deployment on edge hardware. Existing FPGA accelerators often suffer from two main issues: a CPU bottleneck caused by offloading non-convolutional layers, and a lack of versatility due to hand-tuning for specific model versions. SATAY addresses these challenges by providing an automated path to generate hardware that supports diverse operations while maximizing FPGA parallelism.

The SATAY Architecture

SATAY employs a streaming architecture design, where every layer in the network is mapped to a dedicated hardware block. This creates a deep pipeline where data flows continuously through specialized modules, rather than relying on a single general-purpose engine.

Modular Building Blocks

The toolflow utilizes a library of hardware modules for YOLO-specific operations, including Resize, HardSwish, and SPPF (Spatial Pyramid Pooling - Fast). These blocks are composed to form the full detection pipeline on the FPGA fabric.

The SATAY architecture maps YOLO layers to dedicated hardware blocks on the FPGA fabric, supported by modular building blocks like CBS, C3, and SPPF.

Memory Management

To overcome the "memory wall," SATAY stores all model weights directly on the FPGA using BRAM or URAM. This eliminates high-latency external memory accesses during inference. However, long skip connections can consume excessive on-chip memory. SATAY uses an automated algorithm to identify the largest buffers and move them to a "Software FIFO" in external DDR memory. This selective off-chip buffering reduces on-chip usage by up to 56% with minimal impact on bandwidth.

Design Space Exploration

An automated Design Space Exploration (DSE) tool uses a greedy algorithm to allocate Digital Signal Processing (DSP) resources. By identifying and parallelizing bottleneck nodes, the toolflow ensures maximum overall throughput for the generated hardware.

Experimental Results

The SATAY toolflow was evaluated using the COCO-val2017 and COCO-128 datasets across various YOLO versions (v3, v5, v8) and Xilinx FPGA platforms (VCU118, VCU110, U250).

Performance and Efficiency

SATAY-generated designs demonstrate significant performance gains, achieving a 79x speedup over an ARM Cortex-A72 CPU and a 3.6x speedup over a Jetson TX2 GPU. For the YOLOv3-tiny model, SATAY achieved 3.4x higher performance density (GOP/s/DSP) compared to previous state-of-the-art FPGA implementations.

Performance comparison of SATAY-generated designs against prior works, showing the Pareto frontier across mAP and performance density.

State-of-the-Art YOLOv8

This work introduces the first FPGA implementation of YOLOv8 in the literature. The VCU110 variant achieves a 61% mAP at ultra-low latency, defining the current Pareto frontier for high-accuracy, high-performance edge object detection.

Conclusion

SATAY provides a robust, automated solution for deploying the latest YOLO models on FPGAs. By combining a streaming architecture with intelligent memory management, it bridges the gap between model complexity and hardware efficiency. Future work will explore weight binarization and Quantization-Aware Training (QAT) to further reduce memory footprints and expand support to a wider range of edge FPGA devices.

Summary updated

Abstract

AI has led to significant advancements in computer vision and image processing tasks, enabling a wide range of applications in real-life scenarios, from autonomous vehicles to medical imaging. Many of those applications require efficient object detection algorithms and complementary real-time, low latency hardware to perform inference of these algorithms. The YOLO family of models is considered the most efficient for object detection, having only a single model pass. Despite this, the complexity and size of YOLO models can be too computationally demanding for current edge-based platforms. To address this, we present SATAY: a Streaming Architecture Toolflow for Accelerating YOLO. This work tackles the challenges of deploying state-of-the-art object detection models onto FPGA devices for ultralow latency applications, enabling real-time, edge-based object detection. We employ a streaming architecture design for our YOLO accelerators, implementing the complete model on-chip in a deeply pipelined fashion. These accelerators are generated using an automated toolflow, and can target a range of suitable FPGA devices. We introduce novel hardware components to support the operations of YOLO models in a dataflow manner, and off-chip memory buffering to address the limited on-chip memory resources. Our toolflow is able to generate accelerator designs which demonstrate competitive performance and energy characteristics to GPU devices, and which outperform current state-of-the-art FPGA accelerators. The code is available at https://github.com/ICIdsl/satay
BibTeX
@inproceedings{montgomeriecorcoran2023satay,
  author = {Alexander Montgomerie-Corcoran and Petros Toupas and Zhewen Yu and Christos-Savvas Bouganis},
  title = {{SATAY: A Streaming Architecture Toolflow for Accelerating YOLO Models on FPGA Devices}},
  booktitle = {ICFPT},
  year = {2023},
  url = {https://openreview.net/forum?id=tG24zmEFhN},
}
Figure