Realization of Early-Exit Dynamic Neural Networks on Reconfigurable Hardware

Anastasios Dimitriou, Lei Xun, Jonathon S. Hare, Geoff V. MerrettIEEE TCAD 2025 · DOI

Abstract

Deep Neural Networks (DNNs) are traditionally static, processing every input through all layers regardless of complexity, which leads to significant computational waste. This paper presents a comprehensive exploration of early-exit dynamic DNNs implemented on FPGAs. By introducing intermediate classifiers and specialized hardware architectures—specifically pipeline and parallel approaches—the authors demonstrate that FPGAs can bridge the hardware-software gap. Results show that these dynamic models achieve 1.5–3x faster inference and 4x better energy efficiency than embedded GPU platforms.

Introduction

Standard DNNs apply uniform computation to all inputs, failing to distinguish between "easy" and "hard" samples. Early-exiting addresses this by placing intermediate classifiers throughout the network, allowing for early termination once a confidence threshold is reached. While theoretically efficient, the overhead of checking exit conditions on general-purpose hardware often negates these gains. This research addresses this "hardware-software gap" by designing custom FPGA-based accelerators tailored for dynamic branching.

Early-exit DNN architecture with intermediate classifiers

Methodology

The researchers implemented a hardware accelerator using a 2D systolic array on a Xilinx ZCU106 FPGA, employing 8-bit fixed-point representation for efficiency. The core contribution is the development of two distinct hardware execution strategies:

Pipeline Approach

In the pipeline strategy, the backbone network halts while the intermediate classifier runs. If the network is not confident, the backbone resumes. While resource-efficient, this method introduces latency and requires saving intermediate data to off-chip DRAM, which can be a bottleneck for deep networks like VGG19.

Parallel Approach

The parallel approach executes the backbone layer and the intermediate classifier simultaneously. Because the classifier is smaller, it finishes first. If confident, the backbone is terminated; otherwise, it continues without having ever stopped. This eliminates the halting penalty and avoids external DRAM storage by keeping data "in flight."

Comparison of (a) Pipeline and (b) Parallel hardware execution strategies

Key Results and Insights

The models were evaluated on MNIST, CIFAR-10, and CIFAR-100 benchmarks across LeNet-5, AlexNet, VGG19, and ResNet32 architectures.

  • Speed and Efficiency: Dynamic FPGA implementations were 1.5x to 3x faster than static versions and 4x more energy-efficient than the Nvidia Jetson Xavier NX.
  • Architecture Trade-offs: The parallel approach is superior for deep networks where memory movement is costly. Conversely, the pipeline approach is better for simple networks like LeNet-5, where saving hardware area outweighs the small latency penalty.
  • Tunable Performance: By adjusting the confidence threshold (V), the system can adapt to real-time requirements or battery levels at runtime, moving along an accuracy-latency curve.
  • Quantization: The transition to 8-bit fixed-point resulted in a minimal accuracy loss of at most 1.3% compared to full-precision models.

Conclusion

This work demonstrates that custom FPGA architectures can effectively realize the benefits of early-exit DNNs. By hiding exit costs through parallel execution and minimizing memory bottlenecks, FPGAs offer a high-performance, energy-efficient alternative to traditional static execution. Future work includes fully offloading control logic to the FPGA fabric and integrating additional optimizations like pruning and Winograd-GEMM.

Summary updated

BibTeX
@article{dimitriou2025realization,
  author = {Anastasios Dimitriou and Lei Xun and Jonathon S. Hare and Geoff V. Merrett},
  title = {{Realization of Early-Exit Dynamic Neural Networks on Reconfigurable Hardware}},
  journal = {IEEE Trans. Comput. Aided Des. Integr. Circuits Syst.},
  year = {2025},
  doi = {10.1109/tcad.2024.3519055},
}
Figure