HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark

Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, Yingyan Celine Lin

TL;DR

HW-NAS-Bench is the first public benchmark dataset designed to democratize hardware-aware neural architecture search (HW-NAS) by providing comprehensive latency and energy measurements across six distinct hardware platforms. By eliminating the need for complex hardware deployment setups or unreliable proxy metrics like FLOPs, it enables researchers to easily develop and benchmark reproducible, device-specific deep learning models through simple database queries.

What Problem Does This Paper Address?

Deploying deep neural networks (DNNs) to resource-constrained edge devices (like smartphones, smart home devices, or medical implants) requires balancing task accuracy with strict hardware constraints like latency and energy consumption. Hardware-Aware Neural Architecture Search (HW-NAS) automates this design process.

However, HW-NAS faces a massive barrier to entry. It requires cross-disciplinary expertise spanning machine learning algorithms, hardware compilation, and micro-architecture. To estimate hardware costs, researchers must either build complex physical measurement pipelines or rely on oversimplified theoretical metrics like FLOPs (floating-point operations), which frequently fail to reflect real-world performance. Furthermore, benchmarking different HW-NAS algorithms is incredibly difficult and compute-intensive because researchers use different search spaces, hyperparameters, and target hardware.

Figure 1: An illustration of our proposed HW-NAS-Bench

Methods

The authors developed an automated hardware-cost collection pipeline to profile networks across six hardware devices spanning three major categories: commercial edge devices, FPGAs, and ASICs.

  • Commercial Edge Devices: For the NVIDIA Jetson TX2 (Edge GPU), Raspberry Pi 4, Google Edge TPU, and Google Pixel 3, networks were compiled into optimized deployment formats (such as TensorRT and TensorFlow Lite) to measure actual on-device latency and energy consumption.
  • ASIC & FPGA: For specialized hardware like the Eyeriss ASIC accelerator and a Xilinx ZC706 FPGA board, the authors utilized state-of-the-art cycle-accurate simulators (Accelergy + Timeloop, DNN-Chip Predictor) and High-Level Synthesis (Vivado HLS) toolflows to accurately estimate hardware performance.
Figure 2: Illustrating the hardware-cost collection pipeline applicable to various hardware devices.

Datasets

HW-NAS-Bench maps hardware performance metrics directly onto two widely adopted NAS search spaces:

  1. NAS-Bench-201: A cell-based search space containing 46,875 architectures, already mapped to ground-truth accuracy data on CIFAR-10, CIFAR-100, and ImageNet16-120.
  2. FBNet: A massive layer-wise macro-architecture search space containing roughly $10^{21}$ unique candidates. To handle this immense scale, the authors validated a block-sum approximation method (summing the costs of unique network blocks) against real measurements to ensure high accuracy.

Key Insights and Ideas

  • FLOPs are Deceptive: The dataset confirms that theoretical metrics like FLOPs and parameter counts correlate poorly with actual on-device latency and energy consumption (often showing a Kendall Rank Correlation Coefficient below 0.5).
  • Hardware Preferences Vary Drastically: Different hardware architectures favor completely different network structures. A network topology that runs efficiently on an Edge GPU might perform terribly on an ASIC or an Edge TPU due to differences in memory hierarchies, data reuse strategies, and processing elements.
  • Low Cross-Device Correlation: The correlation of hardware costs between different devices can be shockingly low—sometimes near zero. This underscores why a "one-size-fits-all" hardware-optimized network does not exist.
Figure 3: Kendall Rank Correlation Coefficient between real-measured/estimated hardware-cost in different devices considering the NAS-Bench-201 search space.

Key Results

The authors demonstrated the utility of HW-NAS-Bench by running the ProxylessNAS algorithm over the FBNet search space on CIFAR-100. By simply querying the benchmark dataset instead of deploying models to physical hardware, the search was completed with negligible computational overhead.

The experiments quantitatively proved the necessity of device-specific search:

  • An architecture optimized specifically for the Edge GPU achieved a latency of 9.96 ms on that device.
  • An architecture optimized for the Raspberry Pi 4 suffered a 50% latency penalty (13.88 ms) when deployed on the Edge GPU.
  • An architecture optimized for the FPGA suffered a massive 100%+ latency penalty (20.65 ms) when run on the Edge GPU.
Figure 5: Accuracy vs. hardware-cost on different devices considering NAS-Bench-201, where points in red denote the architectures with the optimal trade-offs between “accuracy on ImageNet16- 120 vs. latency measured on Edge GPU”, of which the architectures represent the ground truth of HW-NAS targeting Edge GPUs.

Limitations and Future Work

While comprehensive, the authors note a few limitations in the current release:

  • Edge TPU Energy Data: Because the Google Edge TPU Dev Board lacks an accurate embedded power rail monitor, only latency data (not energy) was collected for this device.
  • Compiler Restrictions: The FBNet search space could not be fully compiled for the Edge TPU due to limitations in Google's proprietary compiler supporting certain layer operations.
  • Future Scope: The authors plan to expand the benchmark to include more diverse search spaces, newer edge hardware (such as dedicated NPUs), and broader computer vision tasks beyond image classification.

Summary updated

Abstract

HardWare-aware Neural Architecture Search (HW-NAS) has recently gained tremendous attention by automating the design of DNNs deployed in more resource-constrained daily life devices. Despite its promising performance, developing optimal HW-NAS solutions can be prohibitively challenging as it requires cross-disciplinary knowledge in the algorithm, micro-architecture, and device-specific compilation. First, to determine the hardware-cost to be incorporated into the NAS process, existing works mostly adopt either pre-collected hardware-cost look-up tables or device-specific hardware-cost models. Both of them limit the development of HW-NAS innovations and impose a barrier-to-entry to non-hardware experts. Second, similar to generic NAS, it can be notoriously difficult to benchmark HW-NAS algorithms due to their significant required computational resources and the differences in adopted search spaces, hyperparameters, and hardware devices. To this end, we develop HW-NAS-Bench, the first public dataset for HW-NAS research which aims to democratize HW-NAS research to non-hardware experts and make HW-NAS research more reproducible and accessible. To design HW-NAS-Bench, we carefully collected the measured/estimated hardware performance of all the networks in the search spaces of both NAS-Bench-201 and FBNet, on six hardware devices that fall into three categories (i.e., commercial edge devices, FPGA, and ASIC). Furthermore, we provide a comprehensive analysis of the collected measurements in HW-NAS-Bench to provide insights for HW-NAS research. Finally, we demonstrate exemplary user cases to (1) show that HW-NAS-Bench allows non-hardware experts to perform HW-NAS by simply querying it and (2) verify that dedicated device-specific HW-NAS can indeed lead to optimal accuracy-cost trade-offs. The codes and all collected data are available at https://github.com/RICE-EIC/HW-NAS-Bench.
BibTeX
@inproceedings{li2021hw,
  author = {Chaojian Li and Zhongzhi Yu and Yonggan Fu and Yongan Zhang and Yang Zhao and Haoran You and Qixuan Yu and Yue Wang and Yingyan Celine Lin},
  title = {{HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=WZ3eDtICG1Kq},
}
Figure