Abstract
Neural Architecture Search (NAS) is often hindered by the high computational cost of evaluating candidate architectures. While predictors can accelerate this process, they typically suffer from a "cold-start" problem when transitioning to new tasks, requiring expensive ground-truth evaluations for retraining. This paper introduces the Transferable Relativistic Predictor (TRP), a framework that leverages cheap, training-free zero-shot metrics for pretraining. By utilizing a smooth ranking loss and an adaptive finetuning strategy based on Chebyshev interpolation, TRP achieves state-of-the-art performance with significantly fewer queries than existing methods.
Introduction
The automation of deep learning model design through Neural Architecture Search (NAS) has seen significant progress, yet the evaluation phase remains a major bottleneck. Predictors—surrogate models that estimate architecture performance—have emerged as a solution. Among these, relativistic predictors, which rank pairs of architectures, have shown superior performance. However, these models face a "cold-start" challenge: they must be retrained from scratch for every new dataset or task, necessitating a large number of fully trained architectures to serve as ground truth.
The Transferable Relativistic Predictor (TRP) addresses this by pretraining on "soft labels" derived from zero-shot metrics. This approach allows the model to develop a "ranking intuition" before seeing any actual performance data from the target task.
Methodology
The TRP framework operates in three distinct stages, as illustrated in Figure 1.
1. Building a Soft-Label Proxy Dataset
Instead of relying on expensive accuracy scores, TRP uses multiple zero-shot metrics (e.g., NWOT, SNIP, Synflow) which are computed without training. Since individual metrics are noisy, TRP fuses them to create soft labels—probabilities that one architecture is superior to another—along with associated confidence scores. This probabilistic approach handles the inherent uncertainty in zero-shot proxies.
2. Pretraining with Smooth Loss and Contrastive Learning
Standard ranking models often use the non-smooth hinge loss, which can be difficult to optimize. TRP introduces a differentiable log-sum-exp approximation to the ranking loss. Furthermore, a contrastive learning objective is integrated to ensure that architectures with similar performance characteristics are mapped to similar regions in the latent representation space, enhancing the model's discriminative power.
3. Adaptive Finetuning via Chebyshev Interpolation
To minimize the number of target-task evaluations, TRP employs an adaptive sampling strategy. It uses Chebyshev interpolation to fit a "marginal benefit curve" of ranking accuracy versus the number of evaluated architectures. Finetuning stops once the curve plateaus, ensuring optimal efficiency.
Experimental Results
TRP was evaluated on major benchmarks including NAS-Bench-201, TransNAS-Bench-101, and the DARTS search space.
- Efficiency and Ranking Accuracy: On NAS-Bench-201, TRP achieved a Kendall’s Tau of 0.8324 with only 30 queries, significantly outperforming previous state-of-the-art methods like Arch-Graph, which reached only 0.6724 with 150 queries.
- Search Performance: In the DARTS search space, TRP identified an architecture with a 2.40% error rate on CIFAR-10 using just 54 evaluations, a task that typically requires hundreds of evaluations for comparable methods.
- Cross-Task Transferability: On TransNAS-Bench-101, TRP demonstrated robust performance across diverse tasks, maintaining a lead of at least 0.22 Kendall’s Tau over competing predictors.
Analysis of Model Components
The effectiveness of TRP's design choices is validated through extensive ablation studies.
As shown in Figure 2, the transition from hinge loss to the smooth log-sum-exp loss (c) consistently improves ranking correlation across datasets. Similarly, the inclusion of contrastive loss (d) provides a substantial boost in performance, confirming that better representation learning is key to transferability. The model also shows stability across a range of hyperparameter values (a, b).
Conclusion
The Transferable Relativistic Predictor effectively mitigates the cold-start problem in NAS by leveraging zero-shot metrics for pretraining. Its combination of smooth ranking loss, contrastive representation learning, and adaptive finetuning allows for highly efficient and accurate architecture search across varied tasks. Future work will focus on more sophisticated information fusion techniques to further enhance the reliability of proxy datasets.
Summary updated