On Latency Predictors for Neural Architecture Search

Yash Akhauri, Mohamed S. Abdelfattah

Abstract

Efficient deployment of neural networks (NN) requires the co-optimization of accuracy and latency. For example, hardware-aware neural architecture search has been used to automatically find NN architectures that satisfy a latency constraint on a specific hardware device. Central to these search algorithms is a prediction model that is designed to provide a hardware latency estimate for a candidate NN architecture. Recent research has shown that the sample efficiency of these predictive models can be greatly improved through pre-training on some training devices with many samples, and then transferring the predictor on the test (target) device. Transfer learning and meta-learning methods have been used for this, but often exhibit significant performance variability. Additionally, the evaluation of existing latency predictors has been largely done on hand-crafted training/test device sets, making it difficult to ascertain design features that compose a robust and general latency predictor. To address these issues, we introduce a comprehensive suite of latency prediction tasks obtained in a principled way through automated partitioning of hardware device sets. We then design a general latency predictor to comprehensively study (1) the predictor architecture, (2) NN sample selection methods, (3) hardware device representations, and (4) NN operation encoding schemes. Building on conclusions from our study, we present an end-to-end latency predictor training strategy that outperforms existing methods on 11 out of 12 difficult latency prediction tasks, improving latency prediction by 22.5% on average, and up to to 87.6% on the hardest tasks. Focusing on latency prediction, our HW-Aware NAS reports a 5.8× speedup in wall-clock time.

Introduction

Deploying deep learning models on edge devices—such as smartphones, IoT devices, or specialized AI accelerators—requires a delicate balance between accuracy and latency. Hardware-aware Neural Architecture Search (NAS) automates this by searching for optimal models for specific chips. However, the bottleneck is often the latency evaluation: measuring thousands of candidate models on real hardware is prohibitively slow and expensive.

To mitigate this, researchers employ latency predictors. While promising, these models often suffer from sample inefficiency and poor generalization to unseen hardware. Furthermore, previous evaluations often relied on "cherry-picked" hardware sets, obscuring the true robustness of proposed methods.

Overview of the NASFLAT framework and its performance compared to baselines.

Methodology: The NASFLAT Framework

We propose NASFLAT (Neural Architecture Sampler And Few-Shot Latency Predictor), which addresses these challenges through four key innovations:

1. Automated Device Partitioning

To ensure rigorous evaluation, we use the Kernighan-Lin algorithm to automatically partition hardware devices into training and test sets based on negative correlation. This creates "adversarial" tasks that test the predictor's ability to generalize across fundamentally different hardware behaviors.

2. GNN Ensemble Architecture

NASFLAT utilizes an ensemble of Graph Convolutional Networks (specifically Dense Graph Flow) and Graph Attention Networks (GAT). This combination allows the model to capture the complex connectivity of neural architectures while avoiding the "over-smoothing" issues common in standard GCNs, preserving the discriminative features necessary for precise latency estimation.

The NASFLAT latency predictor architecture, incorporating hardware and operation embeddings with GNNs.

3. Hardware-Aware Operation Embeddings

Latency is not just a property of an operation (e.g., convolution) but an interaction between the operation and the hardware. NASFLAT concatenates learnable hardware embeddings with operation embeddings. To solve the "cold start" problem for new devices, we initialize target device embeddings using the most similar device from the training pool.

4. Encoding-Based Samplers

Sample selection is critical for few-shot transfer. Instead of random sampling, NASFLAT uses a combination of encodings—CAZ (CATE, Arch2Vec, and Zero-Cost Proxies)—to select a diverse set of "outlier" architectures. This approach significantly reduces prediction variance and improves accuracy with minimal real-world measurements.

Key Results

We validated NASFLAT using large-scale NAS benchmarks (NASBench-201, FBNet) and ground-truth latency data (HW-NAS-Bench, EAGLE) across diverse hardware including CPUs, GPUs, FPGAs, and ASICs.

  • Superior Accuracy: NASFLAT outperformed state-of-the-art methods on 11 out of 12 difficult latency prediction tasks.
  • Adversarial Robustness: On the most challenging hardware sets, NASFLAT improved prediction accuracy by up to 87.6%, with an average improvement of 22.5%.
  • End-to-End Speedup: When integrated into a full NAS pipeline, NASFLAT achieved a 5.8× speedup in wall-clock time compared to existing best-in-class methods.
  • Sample Efficiency: High correlation (Spearman $\rho > 0.9$) is achieved using as few as 20 measured samples on the target device.

Conclusion

NASFLAT demonstrates that robust and efficient latency prediction is achievable through principled device partitioning, sophisticated GNN architectures, and intelligent sample selection. By moving beyond random sampling and hand-crafted benchmarks, we provide a more generalizable framework for hardware-aware NAS. Future work will explore more advanced transfer learning strategies and even more diverse search spaces.

Summary updated

Abstract

Efficient deployment of neural networks (NN) requires the co-optimization of accuracy and latency. For example, hardware-aware neural architecture search has been used to automatically find NN architectures that satisfy a latency constraint on a specific hardware device. Central to these search algorithms is a prediction model that is designed to provide a hardware latency estimate for a candidate NN architecture. Recent research has shown that the sample efficiency of these predictive models can be greatly improved through pre-training on some \textit{training} devices with many samples, and then transferring the predictor on the \textit{test} (target) device. Transfer learning and meta-learning methods have been used for this, but often exhibit significant performance variability. Additionally, the evaluation of existing latency predictors has been largely done on hand-crafted training/test device sets, making it difficult to ascertain design features that compose a robust and general latency predictor. To address these issues, we introduce a comprehensive suite of latency prediction tasks obtained in a principled way through automated partitioning of hardware device sets. We then design a general latency predictor to comprehensively study (1) the predictor architecture, (2) NN sample selection methods, (3) hardware device representations, and (4) NN operation encoding schemes. Building on conclusions from our study, we present an end-to-end latency predictor training strategy that outperforms existing methods on 11 out of 12 difficult latency prediction tasks, improving latency prediction by 22.5\% on average, and up to to 87.6\% on the hardest tasks. Focusing on latency prediction, our HW-Aware NAS reports a $5.8\times$ speedup in wall-clock time. Our code is available on \href{https://github.com/abdelfattah-lab/nasflat_latency}{https://github.com/abdelfattah-lab/nasflat\_latency}.
BibTeX
@inproceedings{akhauri2024latency,
  author = {Yash Akhauri and Mohamed S. Abdelfattah},
  title = {{On Latency Predictors for Neural Architecture Search}},
  booktitle = {MLSys},
  year = {2024},
  url = {https://openreview.net/forum?id=5skNfMH0fK},
}
Figure