NN2CAM: Automated Neural Network Mapping for Multi-Precision Edge Processing on FPGA-Based Cameras

Petar Jokic, Stéphane Emery, Luca BeniniarXiv 2021 · DOI · arXiv

Abstract

The record-breaking achievements of deep neural networks (DNNs) in image classification and detection tasks resulted in a surge of new computer vision applications during the past years. However, their computational complexity is restricting their deployment to powerful stationary or complex dedicated processing hardware, limiting their use in smart edge processing applications. We propose an automated deployment framework for DNN acceleration at the edge on field-programmable gate array (FPGA)-based cameras. The framework automatically converts an arbitrary-sized and quantized trained network into an efficient streaming-processing IP block that is instantiated within a generic adapter block in the FPGA. In contrast to prior work, the accelerator is purely logic and thus supports end-to-end processing on FPGAs without on-chip microprocessors. Our mapping tool features automatic translation from a trained Caffe network, arbitrary layer-wise fixed-point precision for both weights and activations, an efficient XNOR implementation for fully binary layers as well as a balancing mechanism for effective allocation of computational resources in the streaming dataflow. To present the performance of the system we employ this tool to implement two CNN edge processing networks on an FPGA-based high-speed camera with various precision settings showing computational throughputs of up to 337GOPS in low-latency streaming mode (no batching), running entirely on the camera.

Introduction

Deep Neural Networks (DNNs) have revolutionized computer vision, but their "computational hunger" usually confines them to power-hungry GPUs or cloud servers. For "edge" applications—like high-speed industrial sorting or autonomous drones—sending data to the cloud introduces unacceptable latency and privacy risks. While FPGAs are excellent candidates for low-power edge acceleration, existing mapping tools often require specific hardware (like a built-in ARM processor), support only a few fixed precisions, or rely on "batching" multiple images to be efficient, which kills real-time performance. NN2CAM aims to bridge this gap by providing a flexible, automated, and CPU-independent pipeline for deploying CNNs directly onto camera-integrated FPGAs.

The NN2CAM Framework

NN2CAM is a toolchain that automates the path from a trained model to hardware. It interprets a trained model, maps it to a hardware architecture, and instantiates it on the target platform as FPGA firmware.

The NN2CAM automated mapping workflow from trained model to FPGA-based camera firmware.

Key technical pillars include:

Streaming Architecture

Unlike traditional "layer-wise" processing, which finishes one layer before starting the next, NN2CAM uses a streaming (depth-first) approach. Data flows through the layers like water through a pipe, significantly reducing the memory needed to store intermediate results. This allows the camera to start processing the top of an image while the bottom is still being captured by the sensor.

The streaming dataflow architecture used in NN2CAM to minimize latency and memory usage.

Resource Balancing and Multi-Precision Support

Since different layers have different workloads, a slow layer can bottleneck the entire pipeline. NN2CAM includes a script that mathematically balances the allocation of Multiply-Accumulate (MAC) units across layers to ensure a continuous, non-stalling data flow. Furthermore, the framework supports arbitrary layer-wise fixed-point precision. It can handle everything from high-precision 16-bit layers to ultra-efficient "binary" layers (using XNOR logic), allowing developers to trade off accuracy for speed where appropriate.

Pure Logic Implementation

The resulting accelerator is implemented entirely in the FPGA's programmable logic. By removing the dependency on an on-chip microprocessor for inference control, the authors maximize the FPGA fabric available for actual computation and increase portability across different FPGA families, such as the Kintex-7 used in their target "FastEye" camera.

Experimental Evaluation

The researchers evaluated their framework using two primary tasks: M-OCR (Modified MNIST), a version of the classic handwritten digit dataset augmented with noise to simulate industrial environments, and FDDB (Face Detection Data Set and Benchmark), used to evaluate a 9-layer face detection CNN.

The framework was tested on a CSEM FastEye high-speed camera containing a Xilinx Kintex-7 FPGA, yielding the following results:

  • Throughput: Achieved a peak average throughput of 337.8 GOPS (Giga-Operations Per Second) for binary networks.
  • Latency: For a 28x28 OCR task, the latency was a staggering 0.007 ms. Even for a large 640x640 image, the OCR system processed frames in 13.3 ms.
  • Face Detection: A 9-layer mixed-precision CNN achieved 92.3% accuracy on the FDDB benchmark, running at over 40 FPS on 640x640 images. This model used binary weights for the first seven layers to save space, switching to 16-bit for the final layers to maintain high detection accuracy.
  • Efficiency: The entire system (camera + accelerator) consumed roughly 13W of power.

Conclusion and Future Work

NN2CAM demonstrates that high-performance DNN inference is possible at the extreme edge without the need for external CPUs or cloud connectivity. While powerful, the system is currently limited by the physical resources (BRAM and DSP units) of the target FPGA. High-resolution images require large FIFO buffers, which can quickly exhaust internal memory.

Future work will focus on adding support for advanced layer types like residual connections (ResNet) and depth-wise separable convolutions (MobileNet), as well as implementing ternary (2-bit) weights to provide a middle ground between binary and 8-bit precision.

Summary updated

Abstract

The record-breaking achievements of deep neural networks (DNNs) in image classification and detection tasks resulted in a surge of new computer vision applications during the past years. However, their computational complexity is restricting their deployment to powerful stationary or complex dedicated processing hardware, limiting their use in smart edge processing applications. We propose an automated deployment framework for DNN acceleration at the edge on field-programmable gate array (FPGA)-based cameras. The framework automatically converts an arbitrary-sized and quantized trained network into an efficient streaming-processing IP block that is instantiated within a generic adapter block in the FPGA. In contrast to prior work, the accelerator is purely logic and thus supports end-to-end processing on FPGAs without on-chip microprocessors. Our mapping tool features automatic translation from a trained Caffe network, arbitrary layer-wise fixed-point precision for both weights and activations, an efficient XNOR implementation for fully binary layers as well as a balancing mechanism for effective allocation of computational resources in the streaming dataflow. To present the performance of the system we employ this tool to implement two CNN edge processing networks on an FPGA-based high-speed camera with various precision settings showing computational throughputs of up to 337GOPS in low-latency streaming mode (no batching), running entirely on the camera.
BibTeX
@misc{jokic2021nn,
  author = {Petar Jokic and Stéphane Emery and Luca Benini},
  title = {{NN2CAM: Automated Neural Network Mapping for Multi-Precision Edge Processing on FPGA-Based Cameras}},
  year = {2021},
  eprint = {2106.12840},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.12840},
}
Figure