Introduction
Designing deep neural networks for edge devices requires a delicate balance between accuracy and hardware constraints like latency. Hardware-aware Neural Architecture Search (HW-NAS) has emerged as a solution, typically relying on Look-Up Tables (LUTs) or learned latency predictors. However, these methods are often "data-hungry," requiring thousands of measurements on specific target hardware, and they lack generalization—changing the device often means starting the search from scratch.
The Challenge of Hardware Transfer
A significant hurdle in HW-NAS is that architectural performance does not always transfer cleanly between devices. As shown in Figure 1(A), while some performance distributions might correlate between similar devices (like a Pixel 3 and a Raspberry Pi 4), they can fail significantly when moving to specialized hardware like the Eyeriss accelerator. This "one-and-done" nature of traditional search methods makes them expensive and slow to deploy across diverse hardware fleets.
The Sim-is-More Framework
To address these limitations, the authors propose a two-stage framework based on domain randomization and reinforcement learning (RL). The core philosophy, "Sim-is-More," suggests that simulating a wide variety of hardware environments is more valuable for generalization than trying to model a single device perfectly.
Stage 1: Meta-Training on Synthetic Devices
Instead of optimizing for a single real-world device, the RL controller (trained via Proximal Policy Optimization) is exposed to a vast distribution of synthetic devices. These "fake" devices are generated by sampling operation latencies from Gaussian distributions derived from empirical data across 18 different real-world devices. This process forces the controller to learn general strategies for navigating the accuracy-latency trade-off across a wide variety of hardware profiles.
To maintain computational efficiency, the authors utilize FreeREA, a training-free proxy that estimates a network's potential at initialization. This allows the entire meta-training phase to be completed in under one GPU hour on a single NVIDIA RTX 4080.
Stage 2: Zero-Shot Transfer and Adaptation
At test time, the pre-trained controller is deployed on a real target device. Because it has learned a robust policy from diverse synthetic data, it can adapt to the specific characteristics of the new hardware with minimal interaction. Remarkably, the controller requires fewer than 10 real-world latency measurements to identify an optimal architecture for the target device, significantly lower than the hundreds or thousands required by predictor-based methods.
Key Results and Insights
The "Sim-is-More" approach demonstrates that domain randomization is a powerful tool for HW-NAS. Key findings include:
- High Efficiency: The method is extremely lightweight, requiring minimal training time and hardware interaction.
- Robust Adaptability: The agent successfully finds distinct optimal models for diverse hardware, such as FPGAs and mobile CPUs, starting from the same initial state.
- Minimal Overhead: By bypassing the need for extensive pre-collected LUTs or predictors, the framework significantly lowers the barrier to deploying optimized models on new edge hardware.
Limitations and Future Directions
While promising, the study currently focuses on the NATS-Bench search space. Future work will explore larger, more complex spaces like FBNet. Additionally, optimizing the "Goldilocks zone" for the synthetic device distribution—balancing diversity with convergence—remains an open research question. Further analysis is also needed to determine how the controller performs when a target device's latency profile falls completely outside the range of the synthetic training data.
Summary updated