On Latency Predictors for Neural Architecture Search

Yash Akhauri, Mohamed S. Abdelfattah

TL;DR

Hardware-aware Neural Architecture Search (NAS) often relies on few-shot latency predictors to avoid slow on-device profiling, but existing predictors suffer from high variance, unrealistic evaluation benchmarks, and poor cross-device generalization. The authors introduce NASFLAT, an end-to-end latency prediction framework featuring operation-specific hardware embeddings, encoding-based diversity sampling, and realistic low-correlation evaluation splits. NASFLAT outperforms existing predictors on 11 out of 12 benchmark tasks—boosting rank correlation by an average of 22.5% (up to 87.6% on difficult tasks) while speeding up end-to-end hardware-aware NAS by 5.8×.

What Problem Does This Paper Address?

Deploying deep learning models to diverse target hardware (CPUs, GPUs, edge TPUs, microcontrollers) requires balancing predictive accuracy against latency. Profiling every candidate neural network on actual hardware during NAS is prohibitively slow, while proxy metrics like FLOPs or naive layer-lookup tables fail to capture complex runtime behavior such as operator fusion and execution pipelining.

Recent approaches pre-train latency predictors across a pool of source devices and adapt them to a target device using a handful of on-device measurements (few-shot transfer). However, current solutions have critical flaws:

  1. Unrealistic Benchmarks: Source and target device splits in prior benchmarks were hand-picked and highly correlated, masking real-world transfer difficulty.
  2. Brittle Target Sampling: Choosing which architectures to measure on the target device is typically done at random, causing huge predictive variance.
  3. Hardware Agnosticism at the Operator Level: Existing Graph Convolutional Networks (GCNs) treat neural network operations identically across devices, ignoring compiler optimizations and platform-specific kernel implementations.

Methods

The authors introduce NASFLAT (Neural Architecture Sampler And Few-Shot Latency Predictor), combining several targeted design choices:

  • Algorithmic Device Partitioning: The authors use the Kernighan-Lin graph bisection algorithm on device correlation graphs to construct rigorous, low-correlation train/test device splits.
  • Encoding-Based Architecture Sampling: Rather than randomly sampling target architectures or requiring expensive reference latency measurements, NASFLAT clusters candidate networks in latent representation spaces—using unsupervised graph representations (Arch2Vec), transformer-based computational embeddings (CATE), zero-cost accuracy proxies (ZCP), and their concatenation (CAZ). Selecting architectures with low cosine similarity provides a diverse, representative subset using zero target-hardware measurements.
  • Operation-Specific Hardware Embeddings: Instead of appending a single hardware token to the final model representation, NASFLAT concatenates learnable device embeddings directly into each operation embedding before passing them through an operation GNN. This explicitly captures how hardware compilers optimize specific operator sequences.
  • Warm-Start Embedding Initialization: When introduced to a target device, the hardware embedding is initialized using the weights of the most computationally correlated source device rather than initialized randomly.
  • Ensemble GNN Backbone: The latency model combines Dense Graph Flow (DGF) layers (which mitigate over-smoothing via residual connections) with Graph Attention (GAT) modules, feeding into an MLP prediction head augmented by supplementary structural encodings.

Datasets

The framework is evaluated across two standard NAS search spaces using hardware latency measurements from HW-NAS-Bench, HELP, and EAGLE:

  • NASBench-201: Evaluated across standard splits (ND), adversarial splits (NA), and four newly introduced low-correlation algorithmic splits (N1–N4).
  • FBNet: Evaluated on standard splits (FD), adversarial splits (FA), and four new algorithmic splits (F1–F4).

Key Insights and Ideas

  • More pre-training samples can hurt transfer: When source devices are too homogeneous (e.g., exclusively desktop GPUs), scaling up pre-training sample count causes the predictor to overfit to specific hardware traits, degrading target generalization. Diversity across source devices matters more than the raw volume of training samples.
  • Architectural diversity beats random profiling: Selecting target models via cosine distance over structural/computational encodings yields a ~5% boost in rank correlation compared to random sampling, while dramatically reducing cross-trial variance.
  • Compiler effects require local operator-hardware interaction: Modeling device identity as an operation-level interaction improved latency prediction correlation across 11 of 12 device pools (a 7.8% gain), confirming that compilers manipulate adjacent operators differently on different target architectures.

Key Results

  • Predictor Accuracy: NASFLAT achieved state-of-the-art Spearman rank correlation on 11 of 12 device sets, beating prior methods (HELP and MultiPredict) by 22.5% on average and up to 87.6% on the hardest task (F3).
  • Data Efficiency: On NASBench-201, NASFLAT required only 25 source samples per device during pre-training (compared to 900 in HELP and MultiPredict) and 800 samples on FBNet (compared to 4,000), while matching or exceeding target accuracy.
  • NAS Speedup: When integrated with MetaD2A for hardware-aware NAS on CIFAR-100, NASFLAT discovered architectures with superior latency-accuracy Pareto curves while achieving a 5.8× wall-clock speedup in predictor transfer and evaluation. Predictor pre-training required only 25 minutes compared to 25 hours for HELP's meta-learning pipeline.

Limitations and Future Work

  • Encoding Fragility on Giant Search Spaces: CATE underperformed on FBNet because the available profiling dataset (5,000 architectures) was too sparse relative to FBNet's vast design space ($9^{22}$ architectures) to learn reliable computational clusters.
  • Clustering Edge Cases: KMeans occasionally failed to segment the encoding space cleanly, necessitating fallback to cosine similarity metrics.
  • Future Directions: Incorporating more expressive, higher-order meta-learning transfer techniques atop NASFLAT's architecture, and expanding learned graph representations to broader, non-cell-based search spaces.

Summary updated

Abstract

Efficient deployment of neural networks (NN) requires the co-optimization of accuracy and latency. For example, hardware-aware neural architecture search has been used to automatically find NN architectures that satisfy a latency constraint on a specific hardware device. Central to these search algorithms is a prediction model that is designed to provide a hardware latency estimate for a candidate NN architecture. Recent research has shown that the sample efficiency of these predictive models can be greatly improved through pre-training on some \textit{training} devices with many samples, and then transferring the predictor on the \textit{test} (target) device. Transfer learning and meta-learning methods have been used for this, but often exhibit significant performance variability. Additionally, the evaluation of existing latency predictors has been largely done on hand-crafted training/test device sets, making it difficult to ascertain design features that compose a robust and general latency predictor. To address these issues, we introduce a comprehensive suite of latency prediction tasks obtained in a principled way through automated partitioning of hardware device sets. We then design a general latency predictor to comprehensively study (1) the predictor architecture, (2) NN sample selection methods, (3) hardware device representations, and (4) NN operation encoding schemes. Building on conclusions from our study, we present an end-to-end latency predictor training strategy that outperforms existing methods on 11 out of 12 difficult latency prediction tasks, improving latency prediction by 22.5\% on average, and up to to 87.6\% on the hardest tasks. Focusing on latency prediction, our HW-Aware NAS reports a $5.8\times$ speedup in wall-clock time. Our code is available on \href{https://github.com/abdelfattah-lab/nasflat_latency}{https://github.com/abdelfattah-lab/nasflat\_latency}.
BibTeX
@inproceedings{akhauri2024latency,
  author = {Yash Akhauri and Mohamed S. Abdelfattah},
  title = {{On Latency Predictors for Neural Architecture Search}},
  booktitle = {MLSys},
  year = {2024},
  url = {https://openreview.net/forum?id=5skNfMH0fK},
}

Loading related content...

Figure