Abstract
Many existing neural architecture search (NAS) solutions rely on downstream training for architecture evaluation, which takes enormous computations. Considering that these computations bring a large carbon footprint, this paper aims to explore a green (namely environmental-friendly) NAS solution that evaluates architectures without training. Intuitively, gradients, induced by the architecture itself, directly decide the convergence and generalization results. It motivates us to propose the gradient kernel hypothesis: Gradients can be used as a coarse-grained proxy of downstream training to evaluate random-initialized networks. To support the hypothesis, we conduct a theoretical analysis and find a practical gradient kernel that has good correlations with training loss and validation performance. According to this hypothesis, we propose a new kernel based architecture search approach KNAS. Experiments show that KNAS achieves competitive results with orders of magnitude faster than "train-then-test" paradigms on image classification tasks. Furthermore, the extremely low search cost enables its wide applications. The searched network also outperforms strong baseline RoBERTA-large on two text classification tasks.
Introduction
Neural Architecture Search (NAS) has revolutionized deep learning by automating the design of optimal networks. However, it has a massive "dirty secret": it is computationally gluttonous. Traditional "train-then-test" paradigms require training thousands of candidate architectures to evaluate their quality, emitting as much carbon as five cars over their entire lifetimes. While newer "one-shot" or differentiable methods are faster, they often suffer from performance collapse. KNAS addresses this by asking: Can we evaluate an architecture's potential before we even start training it?
The Gradient Kernel Hypothesis
The researchers propose the Gradient Kernel Hypothesis: the gradients induced by a network's architecture at random initialization can serve as a coarse-grained proxy for its eventual downstream performance. The core metric used is the Mean of the Gram Matrix (MGM).
Drawing from Neural Tangent Kernel (NTK) theory, the authors prove that the Frobenius norm of the Gram Matrix determines the upper bound of the training convergence rate. Essentially, a higher MGM indicates a "healthier" gradient flow, suggesting the model will converge faster and reach a lower loss. The MGM captures both gradient magnitude (step size) and gradient correlation (direction consistency).
Experimental Results
KNAS was evaluated across vision (NAS-Bench-201) and language (GLUE) benchmarks. The study shows a high Spearman correlation (around 0.53 to 0.56) between MGM at initialization and actual final validation accuracy, proving that performance at initialization is predictable.
Vision Benchmarks
On NAS-Bench-201, KNAS achieved up to 50x speedups compared to traditional search methods while maintaining competitive accuracy. It outperformed other "training-free" proxies like NASWOT and TE-NAS on ImageNet-16.
NLP Success
When applied to text classification, the KNAS-searched architectures outperformed the strong RoBERTA-large baseline on MRPC (+1.24% accuracy) and RTE (+0.24% accuracy). This demonstrates that the gradient kernel proxy is effective beyond just convolutional neural networks.
Conclusion
KNAS represents a significant step toward making AI research more sustainable. By skipping the heavy lifting of training thousands of models, it identifies high-performing architectures at initialization. While the method currently utilizes a brief fine-grained search to finalize candidates, the underlying hypothesis provides a robust foundation for future training-free NAS research.
Summary updated