SiGeo: Sub-One-Shot NAS via Information Theory and Geometry of Loss Landscape

Hua Zheng, Kuang-Hung Liu, Igor Fedorov, Xin Zhang, Wen-Yen Chen, Wei Wen

Abstract

Neural Architecture Search (NAS) has become a widely used tool for automating neural network design. While one-shot NAS methods have successfully reduced computational requirements, they often require extensive training. On the other hand, zero-shot NAS utilizes training-free proxies to evaluate a candidate architecture's test performance but has two limitations: (1) inability to use the information gained as a network improves with training and (2) unreliable performance, particularly in complex domains like RecSys, due to the multi-modal data inputs and complex architecture configurations. To synthesize the benefits of both methods, we introduce a "sub-one-shot" paradigm that serves as a bridge between zero-shot and one-shot NAS. In sub-one-shot NAS, the supernet is trained using only a small subset of the training data, a phase we refer to as "warm-up." Within this framework, we present SiGeo, a proxy founded on a novel theoretical framework that connects the supernet warm-up with the efficacy of the proxy. Extensive experiments have shown that SiGeo, with the benefit of warm-up, consistently outperforms state-of-the-art NAS proxies on various established NAS benchmarks. When a supernet is warmed up, it can achieve comparable performance to weight-sharing one-shot NAS methods, but with a significant reduction (~60%) in computational costs.

Introduction

Neural Architecture Search (NAS) aims to automate the design of deep learning models, but it faces a classic trade-off between speed and accuracy. One-shot NAS is highly accurate because it trains a "supernet" containing all possible sub-architectures on a full dataset, but it is computationally grueling. Conversely, Zero-shot NAS is incredibly fast, using "training-free" proxies to rank architectures. However, these proxies often ignore the information gained during actual training, making them unreliable—especially in complex domains like Recommendation Systems (RecSys) where data is multi-modal and architectures are heterogeneous.

This paper identifies an "unexplored opportunity": a middle ground where a minimal amount of training, or a "warm-up," can drastically improve proxy reliability without the massive cost of full training.

The Sub-One-Shot Framework

We propose the sub-one-shot paradigm to bridge the gap between zero-shot and one-shot methods. Instead of zero training or full training, the supernet is "warmed up" using a small subset of data (e.g., 1% to 10%). This allows the model weights to escape initial saddle points and move toward a local minimum, making performance indicators significantly clearer.

Comparison of NAS paradigms: One-shot, Sub-one-shot, and Zero-shot.

The SiGeo Proxy and Theoretical Foundation

Within this framework, we introduce SiGeo (Sub-one-shot via Information theory and Geometry of loss landscape). The SiGeo score is a composite metric derived from the geometry of the loss landscape, combining three key components:

  1. Gradient Signal-to-Noise Ratio: Adapted from the SOTA proxy ZiCo, this measures the mean absolute gradient relative to its standard deviation across batches.
  2. Fisher-Rao (FR) Norm: A theoretical measure of a network's generalization capacity. We provide a rigorous theoretical link showing that a higher FR norm during warm-up correlates with lower final training loss.
  3. Current Training Loss: A direct indicator of how well the current architecture is fitting the data.

By weighting these components, SiGeo captures both the potential of an architecture (generalization) and its current state (optimization gap).

Geometry of the loss landscape illustrating the relationship between training loss, optimality gap, and generalization error.

Experimental Results

We validated SiGeo across a wide range of benchmarks, including Computer Vision (NAS-Bench-101, 201, 301) and Recommendation Systems (Criteo, Avazu, KDD Cup 2012).

Key Insights

  • The "Warm-up Magic": Ranking correlation improves dramatically after a tiny bit of training. For example, on NAS-Bench-201, increasing the warm-up level from 0% to 40% improved the Kendall’s tau correlation from 0.58 to 0.70, while purely zero-shot methods remained stagnant.
  • Efficiency Gains: On RecSys benchmarks, SiGeo achieved comparable log-loss performance to one-shot methods but reduced GPU search time by approximately 60% (from ~0.75 days down to ~0.12 days).
  • Superior Accuracy: On CIFAR-10, SiGeo reached 97.01% top-1 accuracy, surpassing other popular proxies like Zen (96.2%) and Synflow (96.1%).

Conclusion

SiGeo demonstrates that the sub-one-shot paradigm offers a superior "sweet spot" for NAS. By leveraging the geometry of the loss landscape after a brief warm-up, SiGeo provides a robust, efficient, and theoretically grounded approach to architecture search that excels even in complex, multi-modal domains like RecSys. Future work will explore automated methods for selecting proxy component weights to further generalize the approach.

Summary updated

Abstract

Neural Architecture Search (NAS) has become a widely used tool for automating neural network design. While one-shot NAS methods have successfully reduced computational requirements, they often require extensive training. On the other hand, zero-shot NAS utilizes training-free proxies to evaluate a candidate architecture's test performance but has two limitations: (1) inability to use the information gained as a network improves with training and (2) unreliable performance, particularly in complex domains like RecSys, due to the multi-modal data inputs and complex architecture configurations. To synthesize the benefits of both methods, we introduce a "sub-one-shot" paradigm that serves as a bridge between zero-shot and one-shot NAS. In sub-one-shot NAS, the supernet is trained using only a small subset of the training data, a phase we refer to as "warm-up." Within this framework, we present SiGeo, a proxy founded on a novel theoretical framework that connects the supernet warm-up with the efficacy of the proxy. Extensive experiments have shown that SiGeo, with the benefit of warm-up, consistently outperforms state-of-the-art NAS proxies on various established NAS benchmarks. When a supernet is warmed up, it can achieve comparable performance to weight-sharing one-shot NAS methods, but with a significant reduction ($\sim 60$\%) in computational costs.
BibTeX
@misc{zheng2023sigeo,
  author = {Hua Zheng and Kuang-Hung Liu and Igor Fedorov and Xin Zhang and Wen-Yen Chen and Wei Wen},
  title = {{SiGeo: Sub-One-Shot NAS via Information Theory and Geometry of Loss Landscape}},
  year = {2023},
  eprint = {2311.13169},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.13169},
}
Figure