GradSign: Model Performance Inference with Theoretical Insights

Zhihao Zhang, Zhihao Jia

Designing high-performing neural networks is increasingly moving away from manual "trial and error" toward Neural Architecture Search (NAS)—an automated process that explores thousands of potential designs. However, the "gold standard" for evaluating a design is to train it to convergence, which is computationally ruinous for large search spaces.

To solve this, researchers use Model Performance Inference (MPI): methods that try to "guess" how well a model will perform without training it. While existing gradient-based proxies (like Snip or Synflow) are fast, they are largely heuristic. They work in practice but lack a rigorous theoretical explanation for why they correlate with final accuracy.

Comparison of MPI Methods

GradSign aims to bridge this gap between speed and theoretical insight. It acts as a "zero-cost" proxy that predicts a neural network's final performance using only a single mini-batch of gradients at initialization. By analyzing the consistency of gradient signs across different data samples, it provides a theoretically grounded metric that significantly accelerates NAS without the need for expensive training.

The Core Innovation: Sample-Wise Optimization

The primary breakthrough of GradSign lies in its analysis of the sample-wise optimization landscape. Instead of looking at the total gradient of a batch, the authors examine the gradients of individual data points.

  1. The Quantity $\Psi$: The authors introduce $\Psi$, a measure of the "density" of local optima for individual samples. Intuitively, if the "best" weight settings for Sample A are very close to the "best" settings for Sample B, the model is easier to optimize and likely to generalize better.
  2. The GradSign Metric: Because $\Psi$ is computationally prohibitive to calculate directly at initialization, the authors propose GradSign as a statistical approximation. It operates by taking a single mini-batch of data and calculating the sign of the gradient for every weight in the network, for every sample.
  3. Consistency is Key: If most samples "agree" on the direction a weight should move (i.e., they have the same gradient sign), the GradSign score is high. The authors mathematically prove that a higher GradSign score—indicating higher sign consistency—corresponds to a lower upper bound on both training and population loss.

Unlike standard methods that require multiple training steps, GradSign requires only one forward and backward pass on a single mini-batch at a random initialization state. Furthermore, while many previous methods focused on the magnitude of gradients (saliency), GradSign suggests that the direction (sign) consistency across samples is a much more robust indicator of a model's optimization potential.

Key Results and Performance

The authors evaluated GradSign across a massive variety of architectures and tasks, including NAS-Bench-101 (423k architectures), NAS-Bench-201, Network Design Spaces (NDS), and ImageNet-1k.

GradSign Performance on NAS-Bench-201
  • Superior Correlation: GradSign consistently outperformed state-of-the-art proxies like Synflow and NASWOT. On NAS-Bench-101, it improved the Spearman’s $\rho$ (a measure of rank correlation) by 25% over previous methods.
  • Better Search Results: When integrated into existing NAS algorithms (such as Evolution or Reinforcement Learning), GradSign-assisted versions discovered models with up to 1.1% higher accuracy than their vanilla counterparts.
  • Unmatched Efficiency: GradSign achieves better predictive power with a single mini-batch than "sample-based" methods (like EconNAS) achieve with 500 mini-batches of proxy training.
  • Robustness: The metric proved effective across different datasets (CIFAR vs. ImageNet) and different architecture types (CNNs vs. search cells), suggesting it captures a generalizable property of neural networks.

Limitations and Future Directions

While GradSign represents a significant step forward, it does have theoretical boundaries. The mathematical proofs rely on the assumption that the optimization landscape is nearly convex and semi-smooth near initialization. While common in deep learning theory, these conditions might be violated by certain non-standard activation functions or extreme architectures. Additionally, the analysis assumes a nearly diagonal Hessian, which may not perfectly capture the dynamics of very deep or narrow networks.

Finally, the authors noted that GradSign scores tend to "cluster" based on architecture classes. Future work could involve adding normalization techniques to better distinguish between models within the same cluster, further refining this powerful zero-cost proxy.

Summary updated

Abstract

A key challenge in neural architecture search (NAS) is quickly inferring the predictive performance of a broad spectrum of networks to discover statistically accurate and computationally efficient ones. We refer to this task as model performance inference (MPI). The current practice for efficient MPI is gradient-based methods that leverage the gradients of a network at initialization to infer its performance. However, existing gradient-based methods rely only on heuristic metrics and lack the necessary theoretical foundations to consolidate their designs. We propose GradSign, an accurate, simple, and flexible metric for model performance inference with theoretical insights. The key idea behind GradSign is a quantity Ψ to analyze the sample-wise optimization landscape of different networks. Theoretically, we show that Ψ is an upper bound for both the training and true population losses of a neural network under reasonable assumptions. However, it is computationally prohibitive to directly calculate Ψ for modern neural networks. To address this challenge, we design GradSign, an accurate and simple approximation of Ψ using the gradients of a network evaluated at a random initialization state. Evaluation on seven NAS benchmarks across three training datasets shows that GradSign generalizes well to real-world networks and consistently outperforms state-of-the-art gradient-based methods for MPI evaluated by Spearman’s ρ and Kendall’s Tau. Additionally, we integrate GradSign into four existing NAS algorithms and show that the GradSign-assisted NAS algorithms outperform their vanilla counterparts by improving the accuracies of best-discovered networks by up to 0.3%, 1.1%, and 1.0% on three real-world tasks. Code is available at https://github.com/JackFram/GradSign
BibTeX
@inproceedings{zhang2022gradsign,
  author = {Zhihao Zhang and Zhihao Jia},
  title = {{GradSign: Model Performance Inference with Theoretical Insights}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=0I5bAghZ-_},
}
Figure