Abstract
A key challenge in neural architecture search (NAS) is quickly inferring the predictive performance of a broad spectrum of networks to discover statistically accurate and computationally efficient ones. We refer to this task as model performance inference (MPI). The current practice for efficient MPI is gradient-based methods that leverage the gradients of a network at initialization to infer its performance. However, existing gradient-based methods rely only on heuristic metrics and lack the necessary theoretical foundations to consolidate their designs. We propose GradSign, an accurate, simple, and flexible metric for model performance inference with theoretical insights. The key idea behind GradSign is a quantity $\Psi$ to analyze the optimization landscape of different networks at the granularity of individual training samples. Theoretically, we show that both the network's training and true population losses are proportionally upper-bounded by $\Psi$ under reasonable assumptions. In addition, we design GradSign, an accurate and simple approximation of $\Psi$ using the gradients of a network evaluated at a random initialization state. Evaluation on seven NAS benchmarks across three training datasets shows that GradSign generalizes well to real-world networks and consistently outperforms state-of-the-art gradient-based methods for MPI evaluated by Spearman's $\rho$ and Kendall's Tau. Additionally, we integrate GradSign into four existing NAS algorithms and show that the GradSign-assisted NAS algorithms outperform their vanilla counterparts by improving the accuracies of best-discovered networks by up to 0.3%, 1.1%, and 1.0% on three real-world tasks.
Introduction
Designing deep learning models is increasingly moving from manual engineering to Neural Architecture Search (NAS). However, the "gold standard" for evaluating a candidate architecture—training it to convergence—is computationally prohibitive when searching through thousands of candidates. To solve this, researchers use Model Performance Inference (MPI) proxies that estimate a model's potential without full training. While gradient-based proxies are fast, most existing ones (like SNIP or Synflow) are based on heuristics rather than firm theory. GradSign bridges this gap by providing a metric that is both computationally simple and theoretically grounded.
Methods: The Sample-Wise Optimization Landscape
GradSign introduces the concept of the sample-wise optimization landscape. Instead of looking at the gradient of the total batch loss, it examines how individual data samples "agree" or "disagree" on the direction weights should move.
The Theoretical Metric ($\Psi$)
The authors define a quantity $\Psi$ (Psi) that measures the density of local optima for individual samples. They mathematically prove that $\Psi$ serves as an upper bound for both training loss and generalization error. If the "best" spots for individual images in a batch are close together, the model is likely to perform better.
The GradSign Approximation
Since calculating $\Psi$ directly is computationally expensive, GradSign serves as a statistical approximation. The algorithm involves:
- Taking a mini-batch of data.
- Calculating the gradient for each sample individually at random initialization.
- For every parameter, summing the signs (positive or negative) of these gradients across the batch.
- The final GradSign score is the sum of the absolute values of these sums.
Intuition: High "sign agreement" implies a smoother optimization landscape where the model can easily find a solution that satisfies all samples simultaneously.
Experimental Results
The authors conducted extensive evaluations across major NAS benchmarks including NAS-Bench-101, NAS-Bench-201, and NDS (Network Design Spaces).
Key findings include:
- Superior Correlation: GradSign consistently achieved higher Spearman’s $\rho$ and Kendall’s Tau than state-of-the-art methods like Synflow and NASWOT. On NAS-Bench-101, it improved the correlation score by 25% over the previous best method.
- Better Architectures: When integrated into standard NAS algorithms, GradSign-assisted versions discovered models with up to 1.1% higher accuracy on CIFAR-100 compared to vanilla counterparts within the same time budget.
- Efficiency: GradSign requires only a single mini-batch and no training, making it orders of magnitude faster than sample-based methods that require proxy training.
Conclusion
GradSign provides a theoretically grounded, training-free metric for NAS that significantly outperforms existing heuristic-based proxies. By focusing on gradient sign consistency, it offers a robust way to identify high-performing architectures at initialization. Future work may explore normalization across architecture classes and the metric's behavior with non-ReLU activation functions.
Summary updated