Revisiting Training-free NAS Metrics: An Efficient Training-based Method

Taojiannan Yang, Linjie Yang, Xiaojie Jin, Chen Chen

Abstract

Recent neural architecture search (NAS) works proposed training-free metrics to rank networks which largely reduced the search cost in NAS. In this paper, we revisit these training-free metrics and find that: (1) the number of parameters (#Param), which is the most straightforward training-free metric, is overlooked in previous works but is surprisingly effective, (2) recent training-free metrics largely rely on the #Param information to rank networks. Our experiments show that the performance of recent training-free metrics drops dramatically when the #Param information is not available. Motivated by these observations, we argue that metrics less correlated with the #Param are desired to provide additional information for NAS. We propose a light-weight training-based metric which has a weak correlation with the #Param while achieving better performance than training-free metrics at a lower search cost. Specifically, on DARTS search space, our method completes searching directly on ImageNet in only 2.6 GPU hours and achieves a top-1/top-5 error rate of 24.1%/7.1%, which is competitive among state-of-the-art NAS methods.

Introduction

Neural Architecture Search (NAS) automates the design of deep learning models but is notoriously resource-intensive, often requiring thousands of GPU hours. To address this, researchers developed "training-free" metrics (like Linear Regions or Neural Tangent Kernels) that estimate a model's performance at initialization. However, this paper identifies a critical oversight: these metrics are highly correlated with the number of parameters (#Param). If a metric primarily acts as a proxy for model size, it fails to provide unique insights into the quality of the architecture itself. This study investigates this "#Param trap" and introduces a more robust, efficient training-based alternative.

The #Param Trap in Training-free Metrics

Analysis of popular training-free metrics like LR (Linear Regions) and NTK (Neural Tangent Kernel) reveals a Kendall’s Tau correlation with parameter count as high as 0.62. When these metrics are tested on groups of models with the same number of parameters, their performance collapses, sometimes performing worse than random search. This suggests that their perceived success in standard benchmarks is largely driven by the inherent correlation between model size and accuracy.

Performance of various NAS metrics across different parameter counts on CIFAR-10, CIFAR-100, and ImageNet16-120.

Methodology: Short-Training NAS (ST-NAS)

To overcome the limitations of static metrics, we propose Short-Training NAS (ST-NAS). This method utilizes a light-weight training-based metric that captures dynamic information with minimal overhead:

  1. Proxy Dataset: A highly condensed version of the target dataset (e.g., 10 classes with 10 images each).
  2. Short Training: Candidate models undergo a brief training phase (e.g., 50 iterations), which is thousands of times faster than full training.
  3. The Angle Metric: We measure the "Angle" between the weights of the final fully-connected (FC) prediction layer before and after this short training. Interestingly, while the angle change in feature extraction layers is positively correlated with accuracy, the change in the prediction layer is negatively correlated. ST-NAS specifically leverages this prediction layer signal.
  4. Combined Ranking: By combining this Angle metric with training loss (AngleLoss), we create a ranking mechanism that is orthogonal to model size.

Experimental Results

ST-NAS was validated across standard benchmarks including NAS-Bench-101, NAS-Bench-201, and the DARTS search space. The results demonstrate that ST-NAS provides a superior balance of search efficiency and accuracy.

Search cost and accuracy comparison of different metrics and their combinations with #Param across benchmarks.

Key findings include:

  • Efficiency: ST-NAS completed a direct search on ImageNet in just 2.6 GPU hours, significantly faster than traditional NAS methods.
  • Accuracy: On ImageNet, the discovered architecture achieved a 24.1% top-1 error rate, competitive with state-of-the-art methods like PC-DARTS and TE-NAS.
  • Robustness: When combined with #Param information, the AngleLoss metric consistently outperformed existing training-free metrics across all tested datasets.

Conclusion and Future Work

This work highlights the over-reliance of current training-free NAS metrics on parameter count. By introducing a light-weight training-based metric, ST-NAS offers a more genuine assessment of architectural potential. Future research should focus on designing search spaces where model size does not dominate performance, allowing for a more rigorous evaluation of how well NAS metrics understand architectural topology.

Summary updated

Abstract

Recent neural architecture search (NAS) works proposed training-free metrics to rank networks which largely reduced the search cost in NAS. In this paper, we revisit these training-free metrics and find that: (1) the number of parameters (\#Param), which is the most straightforward training-free metric, is overlooked in previous works but is surprisingly effective, (2) recent training-free metrics largely rely on the \#Param information to rank networks. Our experiments show that the performance of recent training-free metrics drops dramatically when the \#Param information is not available. Motivated by these observations, we argue that metrics less correlated with the \#Param are desired to provide additional information for NAS. We propose a light-weight training-based metric which has a weak correlation with the \#Param while achieving better performance than training-free metrics at a lower search cost. Specifically, on DARTS search space, our method completes searching directly on ImageNet in only 2.6 GPU hours and achieves a top-1/top-5 error rate of 24.1\%/7.1\%, which is competitive among state-of-the-art NAS methods. Codes are available at \url{https://github.com/taoyang1122/Revisit_TrainingFree_NAS}
BibTeX
@inproceedings{yang2023revisiting,
  author = {Taojiannan Yang and Linjie Yang and Xiaojie Jin and Chen Chen},
  title = {{Revisiting Training-free NAS Metrics: An Efficient Training-based Method}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00473},
}
Figure