Introduction
Efficient Differentiable DNN (EDD) represents a significant shift in how neural networks are optimized for embedded systems. Traditionally, neural architecture search (NAS) and hardware implementation tuning are treated as separate, sequential steps. EDD breaks this paradigm by introducing a framework that simultaneously searches for both the network architecture and its specific hardware implementation parameters—such as quantization bit-widths and parallelism factors—within a single, fully differentiable mathematical formulation. This "fused" approach allows EDD to discover models that are up to 1.45x faster or higher-throughput than state-of-the-art solutions, achieving these results in a remarkably efficient 12-hour search window.
The Challenge: Sequential Optimization vs. Co-Search
In embedded AI development, "one size fits all" is rarely effective. Achieving high performance on specific hardware like mobile GPUs or FPGAs usually requires engineers to first find a suitable model via NAS and then separately optimize its hardware implementation (e.g., tuning bit-widths or memory access).
The authors of EDD argue that this sequential approach is inherently sub-optimal. Even "hardware-aware" NAS typically treats the target hardware as a "black box" with fixed configurations. This ignores a critical synergy: changing hardware implementation details (like using 8-bit instead of 16-bit weights) fundamentally alters which neural architectures are most efficient. EDD addresses this gap by navigating the massive combined design space of software and hardware concurrently.
The EDD Methodology: Fusing Design Spaces
The core innovation of EDD lies in its differentiable formulation, which allows gradient-based optimization to handle both discrete architectural choices and hardware parameters.
The framework employs several key techniques:
- Fused Search Space: Architecture variables (operation types) and hardware variables (quantization, parallel factors, loop tiling) are combined into one space.
- Differentiability via Gumbel-Softmax: To make discrete choices (like 3x3 vs. 5x5 convolutions) compatible with gradient descent, EDD uses the Gumbel-Softmax trick. This enables the system to "propagate" hardware performance requirements directly back into the architecture search.
- Multi-Stage Performance Modeling: Performance (latency/throughput) and resource usage (e.g., FPGA DSP slices) are modeled through four stages, scaling from individual quantized operations up to the entire network.
- Resource Sharing Logic: For FPGA targets where layers might share physical hardware cores, EDD uses a specialized
tanh-based formulation to prevent double-counting resources during the search.
Experimental Results and Benchmarks
The EDD framework was evaluated using the ImageNet dataset. The search was performed on a 100-class subset for efficiency, with final models trained from scratch on the full 1000-class dataset. The authors produced three specialized models: EDD-Net-1 (GPU-optimized), EDD-Net-2 (FPGA-latency optimized), and EDD-Net-3 (FPGA-throughput optimized).
Key performance highlights include:
- GPU Performance (EDD-Net-1): On a Titan RTX, EDD-Net-1 achieved accuracy comparable to Proxyless-GPU while running 1.40x faster (11.17 ms vs 15.72 ms).
- FPGA Latency (EDD-Net-2): Targeted at the Xilinx ZCU102, this model was 1.37x faster than ProxylessNet and 1.53x faster than FBNet implementations on the same hardware.
- FPGA Throughput (EDD-Net-3): On the ZC706, it delivered 1.45x higher throughput (40.2 fps) than the state-of-the-art DNNBuilder solution, while maintaining significantly higher accuracy than VGG16.
Conclusion and Future Directions
EDD demonstrates that making hardware parameters "trainable" allows for unprecedented efficiency. By finding superior models in just 12 GPU hours—compared to the thousands of hours required by reinforcement learning-based NAS—EDD provides a practical path for automated embedded AI design.
While currently focused on GPUs and FPGAs, the methodology is extensible. Future work may include explicit modeling of power consumption and expanding support for dedicated AI ASICs. Additionally, as commercial GPU software stacks evolve to better support mixed-precision computation, the full potential of EDD's mixed-precision search space can be further realized.
Summary updated