Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights

Ondrej Týbl, Lukás NeumannCVPR 2025 · DOI

Abstract

Neural Architecture Search (NAS) typically incurs massive computational costs due to the need for full model training. Training-free proxies offer a solution by scoring architectures at initialization. However, many existing proxies merely correlate with model size rather than architectural quality. We present VKDNW (Variance of Knowledge of Deep Network Weights), a novel proxy based on the Fisher Information Matrix (FIM). VKDNW measures the "balance" of information across weights, providing a size-orthogonal metric that effectively identifies high-performing architectures. Our method achieves state-of-the-art results on standard benchmarks and finds high-accuracy ImageNet models in under half a GPU day.

Introduction

The automation of deep learning model design via Neural Architecture Search (NAS) has traditionally been "astronomically" expensive, often requiring thousands of GPU hours to evaluate candidate architectures. To mitigate this, researchers have turned to "zero-shot" or training-free proxies—mathematical scores calculated at initialization.

A significant challenge in the field is the "trivial proxy" trap: many published proxies primarily correlate with the number of parameters or FLOPs. They essentially suggest that "bigger is better" without distinguishing the merits of specific architectural configurations of the same size. Furthermore, standard evaluation metrics like Kendall’s Tau (KT) can be misleading, as they penalize proxies for failing to rank poor-performing models correctly, even if they excel at identifying the best ones.

Methodology

Variance of Knowledge of Deep Network Weights (VKDNW)

The proposed VKDNW proxy is rooted in Fisher Information theory. The Fisher Information Matrix (FIM) characterizes the sensitivity of a model's output to its parameters. By the Cramér-Rao bound, the inverse of the FIM provides a lower bound on the variance (uncertainty) of weight estimation.

We hypothesize that superior architectures distribute this uncertainty evenly across all weights. VKDNW is calculated as the entropy of the FIM’s eigenvalue spectrum. A higher entropy indicates a more "balanced" network that is likely easier to train and more expressive. To make this calculation tractable for large models, we employ a novel algorithm that estimates the FIM spectrum using a representative subset of parameters, ensuring both efficiency and stability.

A More Robust Evaluation Metric: nDCG

To better assess a proxy's utility in a search process, we advocate for the use of Normalized Discounted Cumulative Gain (nDCG). Unlike correlation coefficients that treat all ranking errors equally, nDCG prioritizes the correct identification and ranking of the top-performing models.

Comparison of ranking metrics; nDCG (right) perfectly identifies the top-performing models unlike standard correlation metrics.

Experimental Results

We evaluated VKDNW on the NAS-Bench-201 benchmark (CIFAR-10, CIFAR-100, ImageNet16-120) and a large-scale MobileNetV2 search space on ImageNet-1K.

Benchmark Performance

On NAS-Bench-201, an aggregated version of our proxy (VKDNW_agg) consistently outperformed all previous training-free methods. Notably, VKDNW demonstrates high discriminatory power, successfully picking "winners" where other proxies like SynFlow or GradNorm struggle.

Comparison of training-free proxies using Normalized Discounted Cumulative Gain (nDCG).

Efficiency and Scalability

In the MobileNetV2 search space, VKDNW identified an architecture achieving 78.8% Top-1 accuracy on ImageNet-1K. The search required only 0.4 GPU days, matching or exceeding the performance of methods that take hundreds of GPU days. Crucially, VKDNW is "orthogonal" to model size; it identifies well-designed architectures regardless of their parameter count. Furthermore, the proxy can be calculated using purely random noise as input, making it truly data-independent.

Discussion and Conclusion

VKDNW addresses key weaknesses in current training-free NAS by providing a size-aware, theoretically grounded proxy. Our analysis also highlights the "trivial proxy" trap, showing that simple metrics like layer counts often outperform complex published methods, underscoring the need for more rigorous evaluation using metrics like nDCG.

Future work will explore more sophisticated parameter selection heuristics to improve FIM estimation stability. Additionally, while VKDNW excels in convolutional spaces, its applicability to non-convolutional architectures, such as Transformers, remains an important area for investigation.

Summary updated

BibTeX
@inproceedings{tybl2025training,
  author = {Ondrej Týbl and Lukás Neumann},
  title = {{Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01386},
}
Figure