FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks

Michaela Blott, Thomas B. Preußer, Nicholas J. Fraser, Giulio Gambardella, Kenneth O'Brien, Yaman Umuroglu, Miriam Leeser, Kees A. VissersACM Trans. Reconfigurable Technol. Syst 2018 · DOI

TL;DR

This paper introduces FINN-R, an end-to-end framework for designing and deploying quantized neural networks (QNNs) on FPGAs. FINN-R automates the exploration of hardware architectures and precisions, enabling significant performance gains and energy efficiency improvements for deep learning inference, even on resource-constrained embedded systems.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are incredibly powerful but come with substantial computational and memory demands. This makes deploying them on edge devices or in real-time applications challenging due to limitations in power, throughput, and latency. While quantization (reducing the precision of weights and activations) offers a promising solution by shrinking model size and computation, exploring the vast design space of QNN architectures and precisions for specific hardware targets is a complex and time-consuming process. This paper addresses the need for a systematic and automated approach to efficiently explore and implement QNNs on FPGAs.

Methods

FINN-R is presented as a modular, end-to-end framework that automates the QNN design and hardware generation process. Key aspects of its methodology include:

  • Quantization-Aware Intermediate Representation (IR): FINN-R uses a directed acyclic graph (DAG) representation that is aware of the quantization of inputs, parameters, and outputs. This allows for quantization-aware optimizations.
  • Modular Frontend/Transform/Backend Structure: Similar to LLVM, FINN-R has distinct stages:
  • Frontend: Converts QNNs from popular frameworks (like Caffe, DarkNet, TensorFlow) into the IR.
  • Transform Passes: Apply optimizations such as direct quantization (converting to fixed-point) and streamlining (absorbing scaling layers into thresholds).
  • Backend: Generates hardware descriptions for specific platforms, supporting two main architectures:
  • Dataflow Architecture (DF): A highly customized, per-layer architecture optimized for a specific network topology and precision.
  • Multilayer Offload Architecture (MO): A more general architecture with larger, homogeneous compute arrays, suitable for resource-constrained devices or very deep networks.
  • Cost Models and Resource Analysis: The framework incorporates detailed cost models for hardware components (LUTs, BRAMs, DSPs) and uses a resource analysis pass to scale operators and optimize performance within given resource budgets.
  • Data Flow Balancing Algorithm: An algorithm that systematically identifies and addresses bottlenecks in a DF pipeline by adjusting concurrency parameters (P, Q, M) to achieve a balanced design.

Datasets

The paper evaluates FINN-R using several standard datasets for training and evaluation:

  • MNIST: Used for training and evaluating MLP-4 networks.
  • CIFAR-10, GTSRB, SVHN: Used for training and evaluating CNV-6 networks.
  • PASCAL VOC: Used for evaluating YOLO-based object detection (Tincy YOLO).
  • ImageNet: Used for evaluating DoReFa-Net/PF.

The paper also references prior work that used these datasets and discusses how their QNN variants were trained.

Key Insights and Ideas

  • Significant Performance Gains from Quantization: The paper strongly demonstrates that moving from floating-point to reduced-precision (even binary) representations can yield substantial increases in performance (e.g., 87x on AWS F1) and reductions in memory footprint.
  • FPGA Suitability for Customization: FPGAs are highlighted as ideal platforms for QNN inference due to their flexibility in implementing custom precisions and architectures, allowing for highly optimized designs.
  • Trade-offs Between Accuracy and Performance: While quantization can lead to a slight drop in accuracy, FINN-R's approach, combined with techniques like quantization-aware training, aims to find optimal trade-offs where accuracy is maintained for the target application.
  • Modular Framework for Design Space Exploration: FINN-R's modularity and automated workflow are crucial for navigating the complex design space of QNNs, enabling researchers and engineers to quickly explore different architectures and precisions.
  • Dataflow vs. Multilayer Offload Architectures: The paper presents two distinct backend architectures (DF and MO) that cater to different design goals: extreme performance and customization (DF) versus minimal footprint and scalability (MO).

Key Results

  • Unprecedented Throughput: FINN-R achieved measured peak performance of 50 TOp/s on AWS F1 and 5 TOp/s on embedded devices (PYNQ-Z1).
  • Performance Scalability: The framework demonstrated performance scaling with FPGA size and achieved significant speedups compared to prior work. For example, on ZC706, FINN-R provided a 10x speedup over dataflow architectures from prior work.
  • Efficiency: The MLP-4 network on the Ultra96 platform achieved 433 GOp/s/W, highlighting excellent energy efficiency.
  • Accuracy Preservation: For many networks, the accuracy drop due to quantization was minimal (e.g., 1.05% for MLP-6, 7.0% for Tincy YOLO, 5.6% for DoReFa-Net/PF compared to unpruned/unquantized equivalents).
  • Resource Utilization: The paper provides detailed breakdowns of LUT, BRAM, and DSP costs for various components and architectures, enabling informed design choices.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while performance estimations for the DF architecture are quite accurate (75%-99%), MO estimations can differ more due to complex external memory transfers and latency modeling.
  • Automated design space exploration is still a work in progress, with the current focus on resource footprint and throughput constraints. Latency and power estimation are left for future work.
  • Retraining is recommended for quantizations below 8 bits, but this pass is not part of the direct quantization process.
  • Future Work:
    • Tightening up performance and resource predictions.
  • Adding support for more network types, such as residual networks and LSTMs.
  • Increasing automation within the framework.
  • Improving accuracy through novel training techniques and quantization methods.
  • The authors are in the process of open-sourcing the framework to foster community contributions.

Summary updated

BibTeX
@article{blott2018finn,
  author = {Michaela Blott and Thomas B. Preußer and Nicholas J. Fraser and Giulio Gambardella and Kenneth O'Brien and Yaman Umuroglu and Miriam Leeser and Kees A. Vissers},
  title = {{FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks}},
  journal = {ACM Trans. Reconfigurable Technol. Syst},
  year = {2018},
  doi = {10.1145/3242897},
}
Figure