ShiftNAS: Improving One-shot NAS via Probability Shift

Mingyang Zhang, Xinyi Yu, Haodong Zhao, Linlin Ou

Introduction

One-shot Neural Architecture Search (NAS) has emerged as a powerful paradigm for designing efficient deep learning models. By training a single "supernet" that encompasses a vast search space of sub-architectures, researchers can theoretically extract optimized models for various hardware constraints without training each from scratch. However, a significant hurdle remains: the performance gap between subnets extracted from the supernet and those trained independently. ShiftNAS addresses this by introducing a "probability shift" mechanism and a differentiable architecture generator, ensuring that models of all sizes—from edge-tier to server-grade—receive sufficient training.

The Problem: The Irwin-Hall Trap

Traditional one-shot NAS methods typically rely on uniform sampling during the supernet training phase. While seemingly fair, this approach falls victim to the Irwin-Hall distribution. When multiple architectural parameters (like layer depth or width) are sampled independently and summed to determine total model complexity, the resulting distribution of model sizes follows a bell curve.

Consequently, the supernet spends the vast majority of its training cycles on "average-sized" models. The extreme ends of the spectrum—the tiny models required for low-power edge devices and the massive models intended for high-performance servers—are sampled infrequently. This leads to undertrained subnets that perform significantly worse than their potential, creating a performance gap that often necessitates expensive retraining.

Methodology

ShiftNAS introduces three core innovations to bridge this gap and streamline the NAS process:

1. Probability Shift Mechanism

To ensure balanced training across the entire search space, ShiftNAS monitors the gradient of the validation loss across different complexity regions. If a specific region (e.g., models around 1.3G FLOPs) exhibits a high gradient, it indicates that the models in that range are currently undertrained. The system then dynamically "shifts" the sampling probability to favor these regions. This automated balancing act ensures that every model size converges effectively, regardless of its position on the Irwin-Hall curve.

2. LSTM-based Architecture Generator (AG)

Rather than relying on random search or evolutionary algorithms after supernet training, ShiftNAS employs an LSTM-based Architecture Generator. This generator is designed to produce architectural sequences that satisfy specific resource constraints (e.g., a 200MB memory limit). Because the AG is integrated into the training process, it learns to identify high-performing configurations much faster than traditional search methods.

3. Differentiable End-to-End Training

The authors utilize the Gumbel Softmax trick and a specialized "matrix mapping" technique to make the architecture sampling process differentiable. This allows the supernet weights and the Architecture Generator to be optimized simultaneously. Furthermore, the method leverages weight entanglement, where smaller operations (like a 3x3 convolution) are nested within larger ones (like a 5x5 convolution), ensuring that improvements in the supernet benefit all potential subnets.

Experimental Results

ShiftNAS was evaluated on the ImageNet-1k classification benchmark and the ADE20k semantic segmentation dataset. It consistently outperformed state-of-the-art NAS frameworks, including AutoFormer, BigNAS, and FocusFormer.

  • Vision Transformers (ViTs): The ShiftFormer-T achieved 76.0% Top-1 accuracy on ImageNet with only 1.3G FLOPs, surpassing AutoFormer-tiny by 1.3%.
  • CNNs: ShiftCNN-B reached 79.6% Top-1 accuracy, outperforming BigNAS-M by 0.7% while requiring significantly less GPU training time.
  • Zero Search/Retrain Cost: A standout feature of ShiftNAS is that its subnets inherit weights directly from the supernet. Unlike many competitors, there is virtually no loss in performance when deploying these subnets immediately, eliminating the need for a secondary search phase or full retraining.

Conclusion and Future Work

ShiftNAS successfully mitigates the sampling bias inherent in one-shot NAS, providing a more robust and efficient path to model deployment. By using gradients as a signal for training sufficiency, it automates the complex task of supernet balancing.

While highly effective for weight-entanglement search spaces and FLOP-based constraints, future work may expand the framework to support "DARTS-like" discrete search spaces and hardware-specific metrics like actual mobile latency. Additionally, the application of "Probability Shift" to the scaling of massive Large Language Models (LLMs) remains a promising avenue for future research.

Summary updated

Abstract

One-shot Neural architecture search (One-shot NAS) has been proposed as a time-efficient approach to obtain optimal subnet architectures and weights under different complexity cases by training only once. However, the subnet performance obtained by weight sharing is often inferior to the performance achieved by retraining. In this paper, we investigate the performance gap and attribute it to the use of uniform sampling, which is a common approach in supernet training. Uniform sampling concentrates training resources on subnets with intermediate computational resources, which are sampled with high probability. However, subnets with different complexity regions require different optimal training strategies for optimal performance.To address the problem of uniform sampling, we propose ShiftNAS, a method that can adjust the sampling probability based on the complexity of subnets. We achieve this by evaluating the performance variation of subnets with different complexity and designing an architecture generator that can accurately and efficiently provide subnets with the desired complexity. Both the sampling probability and the architecture generator can be trained end-to-end in a gradient-based manner. With ShiftNAS, we can directly obtain the optimal model architecture and parameters for a given computational complexity. We evaluate our approach on multiple visual network models, including convolutional neural networks (CNNs) and vision transformers (ViTs), and demonstrate that ShiftNAS is model-agnostic. Experimental results on ImageNet show that ShiftNAS can improve the performance of one-shot NAS without additional consumption. Source codes are available at GitHub.
BibTeX
@inproceedings{zhang2023shiftnas,
  author = {Mingyang Zhang and Xinyi Yu and Haodong Zhao and Linlin Ou},
  title = {{ShiftNAS: Improving One-shot NAS via Probability Shift}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure