Revisiting Training-free NAS Metrics: An Efficient Training-based Method

Taojiannan Yang, Linjie Yang, Xiaojie Jin, Chen Chen

Neural Architecture Search (NAS) has revolutionized deep learning by automating the discovery of high-performing architectures. However, the computational cost of training thousands of candidates remains a significant barrier. Recently, "training-free" metrics have emerged, promising to rank architectures at initialization without any training. This paper, presented at WACV 2023, critically evaluates these metrics and introduces Short-Training NAS (ST-NAS), a more reliable and efficient alternative.

The #Param Trap in Training-free Metrics

The core finding of this research is that many popular training-free metrics, such as Linear Regions (LR) and Neural Tangent Kernel (NTK) proxies, are essentially "over-complicated parameter counters." In standard NAS benchmarks, larger models typically perform better, and these metrics happen to correlate strongly with the number of parameters (#Param).

Correlation between training-free metrics and number of parameters across NAS benchmarks.

As shown in the table above, metrics like LR2 exhibit correlations as high as 0.62 with model size. When architectures of the same size are compared, the predictive power of these metrics often vanishes, rendering them ineffective for fine-grained architectural selection. The challenge, therefore, is to find a metric that provides "orthogonal" information—something that evaluates a model's quality beyond its sheer size.

ST-NAS: A Lightweight Training-based Alternative

To overcome the limitations of training-free proxies, the authors propose ST-NAS. Instead of relying on initialization-only signals, ST-NAS employs a "highly condensed" training phase to gather useful signals.

  1. The Angle Metric: The authors identify that the change in weight direction—the "angle"—between initialization ($W_0$) and a brief training period ($W_t$) is a powerful performance indicator. Crucially, they found that the angle in the final prediction layer (the final fully-connected layer, $\theta_{pred}$) provides a signal that is largely orthogonal to model size. While previous research focused on feature extraction layers, this paper demonstrates that the final layer offers a much cleaner signal for ranking.
  2. Short-Training Scheme: To maintain efficiency, ST-NAS uses a tiny proxy dataset (e.g., 10 classes with 10 images each) and very few iterations (e.g., 50). This process is thousands of times faster than full training, allowing for rapid evaluation.
  3. Combined Ranking: The final ranking is determined by combining the Angle metric with the training loss observed during the short-training phase (AngleLoss).

Experimental Results and Robustness

ST-NAS demonstrates remarkable efficiency and accuracy. On the DARTS search space, it completes a search directly on ImageNet in just 2.6 GPU hours, achieving a competitive 24.1% top-1 error rate. This is significantly faster than early NAS methods that required thousands of hours.

Performance comparison of various metrics across different parameter counts on CIFAR and ImageNet benchmarks.

The figure above illustrates the robustness of the proposed AngleLoss metric. Unlike training-free metrics (LR1, NTK, LR2) which show inconsistent performance when the number of parameters is fixed, AngleLoss (light blue) consistently ranks among the top performers across all parameter scales and datasets. In scenarios where model size was held constant, ST-NAS consistently outperformed training-free metrics, which often performed worse than a simple random baseline.

Conclusion

This study highlights a critical oversight in the development of training-free NAS metrics: their reliance on model size as a proxy for performance. By introducing a lightweight training-based metric that captures architectural quality beyond parameter count, ST-NAS provides a faster and more robust path for automated model design. This work encourages the NAS community to design new search spaces where #Param doesn't dominate performance, allowing for a better evaluation of true architectural innovation.

Summary updated

Abstract

Recent neural architecture search (NAS) works proposed training-free metrics to rank networks which largely reduced the search cost in NAS. In this paper, we revisit these training-free metrics and find that: (1) the number of parameters (#Param), which is the most straightforward training-free metric, is overlooked in previous works but is surprisingly effective, (2) recent training-free metrics largely rely on the #Param information to rank networks. Our experiments show that the performance of recent training-free metrics drops dramatically when the #Param information is not available. Motivated by these observations, we argue that metrics less correlated with the #Param are desired to provide additional information for NAS. We propose a light-weight training-based metric which has a weak correlation with the #Param while achieving better performance than training-free metrics at a lower search cost. Specifically, on DARTS search space, our method completes searching directly on ImageNet in only 2.6 GPU hours and achieves a top-1/top-5 error rate of 24.1%/7.1%, which is competitive among state-of-the-art NAS methods.
BibTeX
@inproceedings{yang2023revisiting,
  author = {Taojiannan Yang and Linjie Yang and Xiaojie Jin and Chen Chen},
  title = {{Revisiting Training-free NAS Metrics: An Efficient Training-based Method}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00473},
}
Figure