Neural Architecture Search (NAS) has revolutionized deep learning by automating the discovery of high-performing architectures. However, the computational cost of training thousands of candidates remains a significant barrier. Recently, "training-free" metrics have emerged, promising to rank architectures at initialization without any training. This paper, presented at WACV 2023, critically evaluates these metrics and introduces Short-Training NAS (ST-NAS), a more reliable and efficient alternative.
The #Param Trap in Training-free Metrics
The core finding of this research is that many popular training-free metrics, such as Linear Regions (LR) and Neural Tangent Kernel (NTK) proxies, are essentially "over-complicated parameter counters." In standard NAS benchmarks, larger models typically perform better, and these metrics happen to correlate strongly with the number of parameters (#Param).
As shown in the table above, metrics like LR2 exhibit correlations as high as 0.62 with model size. When architectures of the same size are compared, the predictive power of these metrics often vanishes, rendering them ineffective for fine-grained architectural selection. The challenge, therefore, is to find a metric that provides "orthogonal" information—something that evaluates a model's quality beyond its sheer size.
ST-NAS: A Lightweight Training-based Alternative
To overcome the limitations of training-free proxies, the authors propose ST-NAS. Instead of relying on initialization-only signals, ST-NAS employs a "highly condensed" training phase to gather useful signals.
- The Angle Metric: The authors identify that the change in weight direction—the "angle"—between initialization ($W_0$) and a brief training period ($W_t$) is a powerful performance indicator. Crucially, they found that the angle in the final prediction layer (the final fully-connected layer, $\theta_{pred}$) provides a signal that is largely orthogonal to model size. While previous research focused on feature extraction layers, this paper demonstrates that the final layer offers a much cleaner signal for ranking.
- Short-Training Scheme: To maintain efficiency, ST-NAS uses a tiny proxy dataset (e.g., 10 classes with 10 images each) and very few iterations (e.g., 50). This process is thousands of times faster than full training, allowing for rapid evaluation.
- Combined Ranking: The final ranking is determined by combining the Angle metric with the training loss observed during the short-training phase (AngleLoss).
Experimental Results and Robustness
ST-NAS demonstrates remarkable efficiency and accuracy. On the DARTS search space, it completes a search directly on ImageNet in just 2.6 GPU hours, achieving a competitive 24.1% top-1 error rate. This is significantly faster than early NAS methods that required thousands of hours.
The figure above illustrates the robustness of the proposed AngleLoss metric. Unlike training-free metrics (LR1, NTK, LR2) which show inconsistent performance when the number of parameters is fixed, AngleLoss (light blue) consistently ranks among the top performers across all parameter scales and datasets. In scenarios where model size was held constant, ST-NAS consistently outperformed training-free metrics, which often performed worse than a simple random baseline.
Conclusion
This study highlights a critical oversight in the development of training-free NAS metrics: their reliance on model size as a proxy for performance. By introducing a lightweight training-based metric that captures architectural quality beyond parameter count, ST-NAS provides a faster and more robust path for automated model design. This work encourages the NAS community to design new search spaces where #Param doesn't dominate performance, allowing for a better evaluation of true architectural innovation.
Summary updated