Transferable Relativistic Predictor: Mitigating Cross-Task Cold-Start Issue in NAS

Nan Li, Bing Xue, Lianbo Ma, Mengjie ZhangIJCAI 2025 · DOI · OpenReview

Abstract

Neural Architecture Search (NAS) is often hindered by the high computational cost of evaluating candidate architectures. While predictors can accelerate this process, they typically suffer from a "cold-start" problem when transitioning to new tasks, requiring expensive ground-truth evaluations for retraining. This paper introduces the Transferable Relativistic Predictor (TRP), a framework that leverages cheap, training-free zero-shot metrics for pretraining. By utilizing a smooth ranking loss and an adaptive finetuning strategy based on Chebyshev interpolation, TRP achieves state-of-the-art performance with significantly fewer queries than existing methods.

Introduction

The automation of deep learning model design through Neural Architecture Search (NAS) has seen significant progress, yet the evaluation phase remains a major bottleneck. Predictors—surrogate models that estimate architecture performance—have emerged as a solution. Among these, relativistic predictors, which rank pairs of architectures, have shown superior performance. However, these models face a "cold-start" challenge: they must be retrained from scratch for every new dataset or task, necessitating a large number of fully trained architectures to serve as ground truth.

The Transferable Relativistic Predictor (TRP) addresses this by pretraining on "soft labels" derived from zero-shot metrics. This approach allows the model to develop a "ranking intuition" before seeing any actual performance data from the target task.

Methodology

The TRP framework operates in three distinct stages, as illustrated in Figure 1.

The TRP framework: (a) Building a soft-label proxy dataset from zero-shot metrics; (b) Pretraining with smooth ranking and contrastive losses; (c) Adaptive finetuning on the target task.

1. Building a Soft-Label Proxy Dataset

Instead of relying on expensive accuracy scores, TRP uses multiple zero-shot metrics (e.g., NWOT, SNIP, Synflow) which are computed without training. Since individual metrics are noisy, TRP fuses them to create soft labels—probabilities that one architecture is superior to another—along with associated confidence scores. This probabilistic approach handles the inherent uncertainty in zero-shot proxies.

2. Pretraining with Smooth Loss and Contrastive Learning

Standard ranking models often use the non-smooth hinge loss, which can be difficult to optimize. TRP introduces a differentiable log-sum-exp approximation to the ranking loss. Furthermore, a contrastive learning objective is integrated to ensure that architectures with similar performance characteristics are mapped to similar regions in the latent representation space, enhancing the model's discriminative power.

3. Adaptive Finetuning via Chebyshev Interpolation

To minimize the number of target-task evaluations, TRP employs an adaptive sampling strategy. It uses Chebyshev interpolation to fit a "marginal benefit curve" of ranking accuracy versus the number of evaluated architectures. Finetuning stops once the curve plateaus, ensuring optimal efficiency.

Experimental Results

TRP was evaluated on major benchmarks including NAS-Bench-201, TransNAS-Bench-101, and the DARTS search space.

  • Efficiency and Ranking Accuracy: On NAS-Bench-201, TRP achieved a Kendall’s Tau of 0.8324 with only 30 queries, significantly outperforming previous state-of-the-art methods like Arch-Graph, which reached only 0.6724 with 150 queries.
  • Search Performance: In the DARTS search space, TRP identified an architecture with a 2.40% error rate on CIFAR-10 using just 54 evaluations, a task that typically requires hundreds of evaluations for comparable methods.
  • Cross-Task Transferability: On TransNAS-Bench-101, TRP demonstrated robust performance across diverse tasks, maintaining a lead of at least 0.22 Kendall’s Tau over competing predictors.

Analysis of Model Components

The effectiveness of TRP's design choices is validated through extensive ablation studies.

Ablation studies for TRP: (a, b) Sensitivity to hyperparameters; (c) Comparison between Hinge and log-sum-exp loss; (d) Impact of contrastive loss.

As shown in Figure 2, the transition from hinge loss to the smooth log-sum-exp loss (c) consistently improves ranking correlation across datasets. Similarly, the inclusion of contrastive loss (d) provides a substantial boost in performance, confirming that better representation learning is key to transferability. The model also shows stability across a range of hyperparameter values (a, b).

Conclusion

The Transferable Relativistic Predictor effectively mitigates the cold-start problem in NAS by leveraging zero-shot metrics for pretraining. Its combination of smooth ranking loss, contrastive representation learning, and adaptive finetuning allows for highly efficient and accurate architecture search across varied tasks. Future work will focus on more sophisticated information fusion techniques to further enhance the reliability of proxy datasets.

Summary updated

Abstract

In neural architecture search (NAS), the relativistic predictor has recently emerged as an attractive technique to solve ranking issue for performance evaluation by predicting the relativistic ranking of architecture pair rather than the absolute performance of an architecture. However, it suffers from a significant cold-start issue, requiring a large amount of evaluated architectures to train an effective predictor on new datasets. In this paper, we propose a transferable relativistic predictor (TRP). Specifically, we construct a proxy dataset using the transferable cheaper-to-obtain performance estimation to softly label the rank between architectural pairs. The soft label with a smooth and easy-to-optimize loss function facilitates the learning of expressive and generalizable representations on the proxy dataset. Furthermore, we construct Chebyshev interpolation for correlation curve to adaptively determine the number of evaluated architectures required on each dataset. Extensive experimental results in different search spaces show the superior performance of TRP compared with state-of-the-art predictors. TRP requires only 54 and 73 evaluated architectures for a warm start on the CIFAR-10 and CIFAR-100 under the DARTS search space.
BibTeX
@inproceedings{li2025transferable,
  author = {Nan Li and Bing Xue and Lianbo Ma and Mengjie Zhang},
  title = {{Transferable Relativistic Predictor: Mitigating Cross-Task Cold-Start Issue in NAS}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/626},
}
Figure