Introduction
One-shot Neural Architecture Search (NAS) has emerged as a powerful paradigm for designing efficient deep learning models. By training a single "supernet" that encompasses a vast search space of sub-architectures, researchers can theoretically extract optimized models for various hardware constraints without training each from scratch. However, a significant hurdle remains: the performance gap between subnets extracted from the supernet and those trained independently. ShiftNAS addresses this by introducing a "probability shift" mechanism and a differentiable architecture generator, ensuring that models of all sizes—from edge-tier to server-grade—receive sufficient training.
The Problem: The Irwin-Hall Trap
Traditional one-shot NAS methods typically rely on uniform sampling during the supernet training phase. While seemingly fair, this approach falls victim to the Irwin-Hall distribution. When multiple architectural parameters (like layer depth or width) are sampled independently and summed to determine total model complexity, the resulting distribution of model sizes follows a bell curve.
Consequently, the supernet spends the vast majority of its training cycles on "average-sized" models. The extreme ends of the spectrum—the tiny models required for low-power edge devices and the massive models intended for high-performance servers—are sampled infrequently. This leads to undertrained subnets that perform significantly worse than their potential, creating a performance gap that often necessitates expensive retraining.
Methodology
ShiftNAS introduces three core innovations to bridge this gap and streamline the NAS process:
1. Probability Shift Mechanism
To ensure balanced training across the entire search space, ShiftNAS monitors the gradient of the validation loss across different complexity regions. If a specific region (e.g., models around 1.3G FLOPs) exhibits a high gradient, it indicates that the models in that range are currently undertrained. The system then dynamically "shifts" the sampling probability to favor these regions. This automated balancing act ensures that every model size converges effectively, regardless of its position on the Irwin-Hall curve.
2. LSTM-based Architecture Generator (AG)
Rather than relying on random search or evolutionary algorithms after supernet training, ShiftNAS employs an LSTM-based Architecture Generator. This generator is designed to produce architectural sequences that satisfy specific resource constraints (e.g., a 200MB memory limit). Because the AG is integrated into the training process, it learns to identify high-performing configurations much faster than traditional search methods.
3. Differentiable End-to-End Training
The authors utilize the Gumbel Softmax trick and a specialized "matrix mapping" technique to make the architecture sampling process differentiable. This allows the supernet weights and the Architecture Generator to be optimized simultaneously. Furthermore, the method leverages weight entanglement, where smaller operations (like a 3x3 convolution) are nested within larger ones (like a 5x5 convolution), ensuring that improvements in the supernet benefit all potential subnets.
Experimental Results
ShiftNAS was evaluated on the ImageNet-1k classification benchmark and the ADE20k semantic segmentation dataset. It consistently outperformed state-of-the-art NAS frameworks, including AutoFormer, BigNAS, and FocusFormer.
- Vision Transformers (ViTs): The ShiftFormer-T achieved 76.0% Top-1 accuracy on ImageNet with only 1.3G FLOPs, surpassing AutoFormer-tiny by 1.3%.
- CNNs: ShiftCNN-B reached 79.6% Top-1 accuracy, outperforming BigNAS-M by 0.7% while requiring significantly less GPU training time.
- Zero Search/Retrain Cost: A standout feature of ShiftNAS is that its subnets inherit weights directly from the supernet. Unlike many competitors, there is virtually no loss in performance when deploying these subnets immediately, eliminating the need for a secondary search phase or full retraining.
Conclusion and Future Work
ShiftNAS successfully mitigates the sampling bias inherent in one-shot NAS, providing a more robust and efficient path to model deployment. By using gradients as a signal for training sufficiency, it automates the complex task of supernet balancing.
While highly effective for weight-entanglement search spaces and FLOP-based constraints, future work may expand the framework to support "DARTS-like" discrete search spaces and hardware-specific metrics like actual mobile latency. Additionally, the application of "Probability Shift" to the scaling of massive Large Language Models (LLMs) remains a promising avenue for future research.
Summary updated