Sim-is-More: Randomizing HW-NAS with Synthetic Devices

Francesco Capuano, Gabriele Tiboni, Niccolò Cavagnero, Giuseppe AvertaarXiv 2025 · DOI · arXiv

Introduction

Designing deep neural networks for edge devices requires a delicate balance between accuracy and hardware constraints like latency. Hardware-aware Neural Architecture Search (HW-NAS) has emerged as a solution, typically relying on Look-Up Tables (LUTs) or learned latency predictors. However, these methods are often "data-hungry," requiring thousands of measurements on specific target hardware, and they lack generalization—changing the device often means starting the search from scratch.

The Challenge of Hardware Transfer

A significant hurdle in HW-NAS is that architectural performance does not always transfer cleanly between devices. As shown in Figure 1(A), while some performance distributions might correlate between similar devices (like a Pixel 3 and a Raspberry Pi 4), they can fail significantly when moving to specialized hardware like the Eyeriss accelerator. This "one-and-done" nature of traditional search methods makes them expensive and slow to deploy across diverse hardware fleets.

The Sim-is-More framework: (A) demonstrates the difficulty of direct transfer between real devices, while (B) outlines the proposed two-stage approach using synthetic device distributions.

The Sim-is-More Framework

To address these limitations, the authors propose a two-stage framework based on domain randomization and reinforcement learning (RL). The core philosophy, "Sim-is-More," suggests that simulating a wide variety of hardware environments is more valuable for generalization than trying to model a single device perfectly.

Stage 1: Meta-Training on Synthetic Devices

Instead of optimizing for a single real-world device, the RL controller (trained via Proximal Policy Optimization) is exposed to a vast distribution of synthetic devices. These "fake" devices are generated by sampling operation latencies from Gaussian distributions derived from empirical data across 18 different real-world devices. This process forces the controller to learn general strategies for navigating the accuracy-latency trade-off across a wide variety of hardware profiles.

To maintain computational efficiency, the authors utilize FreeREA, a training-free proxy that estimates a network's potential at initialization. This allows the entire meta-training phase to be completed in under one GPU hour on a single NVIDIA RTX 4080.

Stage 2: Zero-Shot Transfer and Adaptation

At test time, the pre-trained controller is deployed on a real target device. Because it has learned a robust policy from diverse synthetic data, it can adapt to the specific characteristics of the new hardware with minimal interaction. Remarkably, the controller requires fewer than 10 real-world latency measurements to identify an optimal architecture for the target device, significantly lower than the hundreds or thousands required by predictor-based methods.

Key Results and Insights

The "Sim-is-More" approach demonstrates that domain randomization is a powerful tool for HW-NAS. Key findings include:

  • High Efficiency: The method is extremely lightweight, requiring minimal training time and hardware interaction.
  • Robust Adaptability: The agent successfully finds distinct optimal models for diverse hardware, such as FPGAs and mobile CPUs, starting from the same initial state.
  • Minimal Overhead: By bypassing the need for extensive pre-collected LUTs or predictors, the framework significantly lowers the barrier to deploying optimized models on new edge hardware.

Limitations and Future Directions

While promising, the study currently focuses on the NATS-Bench search space. Future work will explore larger, more complex spaces like FBNet. Additionally, optimizing the "Goldilocks zone" for the synthetic device distribution—balancing diversity with convergence—remains an open research question. Further analysis is also needed to determine how the controller performs when a target device's latency profile falls completely outside the range of the synthetic training data.

Summary updated

Abstract

Existing hardware-aware NAS (HW-NAS) methods typically assume access to precise information circa the target device, either via analytical approximations of the post-compilation latency model, or through learned latency predictors. Such approximate approaches risk introducing estimation errors that may prove detrimental in risk-sensitive applications. In this work, we propose a two-stage HW-NAS framework, in which we first learn an architecture controller on a distribution of synthetic devices, and then directly deploy the controller on a target device. At test-time, our network controller deploys directly to the target device without relying on any pre-collected information, and only exploits direct interactions. In particular, the pre-training phase on synthetic devices enables the controller to design an architecture for the target device by interacting with it through a small number of high-fidelity latency measurements. To guarantee accessibility of our method, we only train our controller with training-free accuracy proxies, allowing us to scale the meta-training phase without incurring the overhead of full network training. We benchmark on HW-NATS-Bench, demonstrating that our method generalizes to unseen devices and searches for latency-efficient architectures by in-context adaptation using only a few real-world latency evaluations at test-time.
BibTeX
@misc{capuano2025sim,
  author = {Francesco Capuano and Gabriele Tiboni and Niccolò Cavagnero and Giuseppe Averta},
  title = {{Sim-is-More: Randomizing HW-NAS with Synthetic Devices}},
  year = {2025},
  eprint = {2504.00663},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.00663},
}
Figure