Abstract
HardWare-aware Neural Architecture Search (HW-NAS) has recently gained tremendous attention by automating the design of DNNs deployed in more resource-constrained daily life devices. Despite its promising performance, developing optimal HW-NAS solutions can be prohibitively challenging as it requires cross-disciplinary knowledge in the algorithm, micro-architecture, and device-specific compilation. First, to determine the hardware-cost to be incorporated into the NAS process, existing works mostly adopt either pre-collected hardware-cost look-up tables or device-specific hardware-cost models. Both of them limit the development of HW-NAS innovations and impose a barrier-to-entry to non-hardware experts. Second, similar to generic NAS, it can be notoriously difficult to benchmark HW-NAS algorithms due to their significant required computational resources and the differences in adopted search spaces, hyperparameters, and hardware devices. To this end, we develop HW-NAS-Bench, the first public dataset for HW-NAS research which aims to democratize HW-NAS research to non-hardware experts and make HW-NAS research more reproducible and accessible. To design HW-NAS-Bench, we carefully collected the measured/estimated hardware performance of all the networks in the search spaces of both NAS-Bench-201 and FBNet, on six hardware devices that fall into three categories (i.e., commercial edge devices, FPGA, and ASIC). Furthermore, we provide a comprehensive analysis of the collected measurements in HW-NAS-Bench to provide insights for HW-NAS research. Finally, we demonstrate exemplary user cases to (1) show that HW-NAS-Bench allows non-hardware experts to perform HW-NAS by simply querying it and (2) verify that dedicated device-specific HW-NAS can indeed lead to optimal accuracy-cost trade-offs.
Introduction
Neural Architecture Search (NAS) has revolutionized how we design deep learning models, but a model that runs efficiently on a massive server GPU might perform poorly on a mobile phone or an embedded FPGA. This realization birthed Hardware-aware NAS (HW-NAS), which treats hardware efficiency—such as latency and energy consumption—as a first-class citizen alongside accuracy.
However, HW-NAS is notoriously difficult to research. It requires a rare "full-stack" expertise spanning deep learning algorithms, hardware compilation, and low-level performance measurement. To bridge this gap, HW-NAS-Bench provides a large-scale public dataset that removes the need for expensive hardware setups or specialized engineering expertise.
The HW-NAS-Bench Solution
Developing efficient AI for the "edge" is a balancing act between accuracy, battery life (energy), and real-time requirements (latency). The authors identify two major barriers in the current landscape: the Expertise Barrier, where researchers must manage complex hardware compilation pipelines, and the Reproducibility Crisis, caused by inconsistent hardware and search spaces across different studies.
HW-NAS-Bench addresses these by creating a comprehensive "lookup table" for hardware performance across two representative search spaces:
- NAS-Bench-201: A cell-based space containing 15,625 unique architectures.
- FBNet: A layer-wise space designed for hardware friendliness, containing a staggering $10^{21}$ possible architectures.
Methodology and Hardware Platforms
The researchers deployed these architectures across six hardware devices representing three major categories:
- Commercial Edge Devices: NVIDIA Jetson TX2 (Edge GPU), Raspberry Pi 4, and Google Pixel 3.
- ASIC: Google Edge TPU (measured) and the Eyeriss accelerator (estimated via simulators like Timeloop).
- FPGA: Xilinx ZC706 (estimated via a Vivado HLS-based pipeline).
For the massive FBNet space, they used a validated approximation method: measuring the cost of unique "blocks" and summing them to estimate total network cost, maintaining high correlation with real-world measurements.
Key Insights: The "FLOPs" Trap and Device Diversity
A critical finding of the study is that theoretical metrics like FLOPs and parameter counts are often poor predictors of actual runtime. In many cases, the correlation between FLOPs and actual latency on devices like Edge GPUs or TPUs was near zero.
Furthermore, the benchmark reveals that hardware is not created equal. An architecture optimized for an Edge GPU might be highly inefficient on an FPGA. As shown in the correlation matrices below, the relationship between latency across different devices can be surprisingly low, highlighting the necessity of device-specific search.
Results and Impact
Using the benchmark with standard algorithms like ProxylessNAS, the authors demonstrated that targeting a specific device leads to significantly better results. For example, a model searched specifically for an Edge GPU achieved 9.96ms latency, whereas a model searched for an FPGA but run on the GPU took 20.65ms—more than double the time for the same accuracy level.
By providing a pre-measured dataset, HW-NAS-Bench democratizes the field. Researchers can now run hardware-aware searches on a standard laptop in minutes by querying the API, turning HW-NAS from a niche hardware engineering task into an accessible and reproducible algorithmic challenge.
Summary updated