EDD: Efficient Differentiable DNN Architecture and Implementation Co-search for Embedded AI Solutions

Yuhong Li, Cong Hao, Xiaofan Zhang, Xinheng Liu, Yao Chen, Jinjun Xiong, Wen-mei W. Hwu, Deming Chen

Introduction

Efficient Differentiable DNN (EDD) represents a significant shift in how neural networks are optimized for embedded systems. Traditionally, neural architecture search (NAS) and hardware implementation tuning are treated as separate, sequential steps. EDD breaks this paradigm by introducing a framework that simultaneously searches for both the network architecture and its specific hardware implementation parameters—such as quantization bit-widths and parallelism factors—within a single, fully differentiable mathematical formulation. This "fused" approach allows EDD to discover models that are up to 1.45x faster or higher-throughput than state-of-the-art solutions, achieving these results in a remarkably efficient 12-hour search window.

In embedded AI development, "one size fits all" is rarely effective. Achieving high performance on specific hardware like mobile GPUs or FPGAs usually requires engineers to first find a suitable model via NAS and then separately optimize its hardware implementation (e.g., tuning bit-widths or memory access).

The authors of EDD argue that this sequential approach is inherently sub-optimal. Even "hardware-aware" NAS typically treats the target hardware as a "black box" with fixed configurations. This ignores a critical synergy: changing hardware implementation details (like using 8-bit instead of 16-bit weights) fundamentally alters which neural architectures are most efficient. EDD addresses this gap by navigating the massive combined design space of software and hardware concurrently.

The EDD Methodology: Fusing Design Spaces

The core innovation of EDD lies in its differentiable formulation, which allows gradient-based optimization to handle both discrete architectural choices and hardware parameters.

Overview of the EDD framework showing the fused search space and multi-stage performance modeling.

The framework employs several key techniques:

  1. Fused Search Space: Architecture variables (operation types) and hardware variables (quantization, parallel factors, loop tiling) are combined into one space.
  2. Differentiability via Gumbel-Softmax: To make discrete choices (like 3x3 vs. 5x5 convolutions) compatible with gradient descent, EDD uses the Gumbel-Softmax trick. This enables the system to "propagate" hardware performance requirements directly back into the architecture search.
  3. Multi-Stage Performance Modeling: Performance (latency/throughput) and resource usage (e.g., FPGA DSP slices) are modeled through four stages, scaling from individual quantized operations up to the entire network.
  4. Resource Sharing Logic: For FPGA targets where layers might share physical hardware cores, EDD uses a specialized tanh-based formulation to prevent double-counting resources during the search.

Experimental Results and Benchmarks

The EDD framework was evaluated using the ImageNet dataset. The search was performed on a 100-class subset for efficiency, with final models trained from scratch on the full 1000-class dataset. The authors produced three specialized models: EDD-Net-1 (GPU-optimized), EDD-Net-2 (FPGA-latency optimized), and EDD-Net-3 (FPGA-throughput optimized).

Comparison of EDD-Net models against baseline and hardware-aware NAS models on GPU and FPGA latency.

Key performance highlights include:

  • GPU Performance (EDD-Net-1): On a Titan RTX, EDD-Net-1 achieved accuracy comparable to Proxyless-GPU while running 1.40x faster (11.17 ms vs 15.72 ms).
  • FPGA Latency (EDD-Net-2): Targeted at the Xilinx ZCU102, this model was 1.37x faster than ProxylessNet and 1.53x faster than FBNet implementations on the same hardware.
  • FPGA Throughput (EDD-Net-3): On the ZC706, it delivered 1.45x higher throughput (40.2 fps) than the state-of-the-art DNNBuilder solution, while maintaining significantly higher accuracy than VGG16.

Conclusion and Future Directions

EDD demonstrates that making hardware parameters "trainable" allows for unprecedented efficiency. By finding superior models in just 12 GPU hours—compared to the thousands of hours required by reinforcement learning-based NAS—EDD provides a practical path for automated embedded AI design.

While currently focused on GPUs and FPGAs, the methodology is extensible. Future work may include explicit modeling of power consumption and expanding support for dedicated AI ASICs. Additionally, as commercial GPU software stacks evolve to better support mixed-precision computation, the full potential of EDD's mixed-precision search space can be further realized.

Summary updated

BibTeX
@inproceedings{li2020edd,
  author = {Yuhong Li and Cong Hao and Xiaofan Zhang and Xinheng Liu and Yao Chen and Jinjun Xiong and Wen-mei W. Hwu and Deming Chen},
  title = {{EDD: Efficient Differentiable DNN Architecture and Implementation Co-search for Embedded AI Solutions}},
  booktitle = {DAC},
  year = {2020},
  doi = {10.1109/dac18072.2020.9218749},
}
Figure