KNAS: Green Neural Architecture Search

Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, Hongxia Yang

Abstract

Many existing neural architecture search (NAS) solutions rely on downstream training for architecture evaluation, which takes enormous computations. Considering that these computations bring a large carbon footprint, this paper aims to explore a green (namely environmental-friendly) NAS solution that evaluates architectures without training. Intuitively, gradients, induced by the architecture itself, directly decide the convergence and generalization results. It motivates us to propose the gradient kernel hypothesis: Gradients can be used as a coarse-grained proxy of downstream training to evaluate random-initialized networks. To support the hypothesis, we conduct a theoretical analysis and find a practical gradient kernel that has good correlations with training loss and validation performance. According to this hypothesis, we propose a new kernel based architecture search approach KNAS. Experiments show that KNAS achieves competitive results with orders of magnitude faster than "train-then-test" paradigms on image classification tasks. Furthermore, the extremely low search cost enables its wide applications. The searched network also outperforms strong baseline RoBERTA-large on two text classification tasks.

Introduction

Neural Architecture Search (NAS) has revolutionized deep learning by automating the design of optimal networks. However, it has a massive "dirty secret": it is computationally gluttonous. Traditional "train-then-test" paradigms require training thousands of candidate architectures to evaluate their quality, emitting as much carbon as five cars over their entire lifetimes. While newer "one-shot" or differentiable methods are faster, they often suffer from performance collapse. KNAS addresses this by asking: Can we evaluate an architecture's potential before we even start training it?

The Gradient Kernel Hypothesis

The researchers propose the Gradient Kernel Hypothesis: the gradients induced by a network's architecture at random initialization can serve as a coarse-grained proxy for its eventual downstream performance. The core metric used is the Mean of the Gram Matrix (MGM).

Drawing from Neural Tangent Kernel (NTK) theory, the authors prove that the Frobenius norm of the Gram Matrix determines the upper bound of the training convergence rate. Essentially, a higher MGM indicates a "healthier" gradient flow, suggesting the model will converge faster and reach a lower loss. The MGM captures both gradient magnitude (step size) and gradient correlation (direction consistency).

Correlation between MGM Rank and Validation Accuracy. Higher MGM ranks consistently correspond to higher validation accuracy, supporting the Gradient Kernel Hypothesis.

Experimental Results

KNAS was evaluated across vision (NAS-Bench-201) and language (GLUE) benchmarks. The study shows a high Spearman correlation (around 0.53 to 0.56) between MGM at initialization and actual final validation accuracy, proving that performance at initialization is predictable.

Vision Benchmarks

On NAS-Bench-201, KNAS achieved up to 50x speedups compared to traditional search methods while maintaining competitive accuracy. It outperformed other "training-free" proxies like NASWOT and TE-NAS on ImageNet-16.

MGM Rank vs. Test Accuracy on CIFAR-100 and ImageNet-16. The positive correlation demonstrates the effectiveness of MGM as a proxy across different datasets.

NLP Success

When applied to text classification, the KNAS-searched architectures outperformed the strong RoBERTA-large baseline on MRPC (+1.24% accuracy) and RTE (+0.24% accuracy). This demonstrates that the gradient kernel proxy is effective beyond just convolutional neural networks.

Conclusion

KNAS represents a significant step toward making AI research more sustainable. By skipping the heavy lifting of training thousands of models, it identifies high-performing architectures at initialization. While the method currently utilizes a brief fine-grained search to finalize candidates, the underlying hypothesis provides a robust foundation for future training-free NAS research.

Summary updated

Abstract

Many existing neural architecture search (NAS) solutions rely on downstream training for architecture evaluation, which takes enormous computations. Considering that these computations bring a large carbon footprint, this paper aims to explore a green (namely environmental-friendly) NAS solution that evaluates architectures without training. Intuitively, gradients, induced by the architecture itself, directly decide the convergence and generalization results. It motivates us to propose the gradient kernel hypothesis: Gradients can be used as a coarse-grained proxy of downstream training to evaluate random-initialized networks. To support the hypothesis, we conduct a theoretical analysis and find a practical gradient kernel that has good correlations with training loss and validation performance. According to this hypothesis, we propose a new kernel based architecture search approach KNAS. Experiments show that KNAS achieves competitive results with orders of magnitude faster than "train-then-test" paradigms on image classification tasks. Furthermore, the extremely low search cost enables its wide applications. The searched network also outperforms strong baseline RoBERTA-large on two text classification tasks. Codes are available at \url{https://github.com/Jingjing-NLP/KNAS} .
BibTeX
@inproceedings{xu2021knas,
  author = {Jingjing Xu and Liang Zhao and Junyang Lin and Rundong Gao and Xu Sun and Hongxia Yang},
  title = {{KNAS: Green Neural Architecture Search}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=HudZhXB4mB},
}
Figure