Semi-Supervised Neural Architecture Search

Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, Tie-Yan Liu

Abstract

Neural architecture search (NAS) relies on a good controller to generate better architectures or predict the accuracy of given architectures. However, training the controller requires both abundant and high-quality pairs of architectures and their accuracy, while it is costly to evaluate an architecture and obtain its accuracy. In this paper, we propose SemiNAS, a semi-supervised NAS approach that leverages numerous unlabeled architectures (without evaluation and thus nearly no cost). Specifically, SemiNAS 1) trains an initial accuracy predictor with a small set of architecture-accuracy data pairs; 2) uses the trained accuracy predictor to predict the accuracy of large amount of architectures (without evaluation); and 3) adds the generated data pairs to the original data to further improve the predictor. The trained accuracy predictor can be applied to various NAS algorithms by predicting the accuracy of candidate architectures for them. SemiNAS has two advantages: 1) It reduces the computational cost under the same accuracy guarantee. On NASBench-101 benchmark dataset, it achieves comparable accuracy with gradient-based method while using only 1/7 architecture-accuracy pairs. 2) It achieves higher accuracy under the same computational cost. It achieves 94.02% test accuracy on NASBench-101, outperforming all the baselines when using the same number of architectures. On ImageNet, it achieves 23.5% top-1 error rate (under 600M FLOPS constraint) using 4 GPU-days for search. We further apply it to LJSpeech text to speech task and it achieves 97% intelligibility rate in the low-resource setting and 15% test error rate in the robustness setting, with 9%, 7% improvements over the baseline respectively.

Introduction

Neural Architecture Search (NAS) aims to automate the design of deep learning models, but it faces a massive "evaluator bottleneck." To find a high-performing architecture, a controller needs to understand which designs work best. However, determining the ground-truth accuracy of a single architecture typically requires training it from scratch, which is computationally ruinous—often costing hundreds or thousands of GPU days.

While "one-shot" NAS methods use weight-sharing to speed up this process, they often suffer from poor ranking quality; the evaluator might incorrectly rank architectures due to the noise introduced by shared weights. SemiNAS addresses this by asking: Can we use the millions of architectures we haven't tested to help us better understand the few we have?

Methodology

SemiNAS integrates semi-supervised learning into the NAS pipeline by building upon the Neural Architecture Optimization (NAO) framework. It follows a iterative three-step cycle:

  1. Initial Training: An accuracy predictor (comprising an LSTM encoder, a Predictor MLP, and an LSTM decoder) is trained using a small set of "labeled" data—architectures that have been fully trained and evaluated.
  2. Pseudo-Labeling: This initial predictor estimates the accuracy of a much larger set of "unlabeled" architectures, which are computationally free to generate but have not been trained.
  3. Semi-Supervised Refinement: The predictor is retrained using both the small ground-truth dataset and the large pseudo-labeled dataset.

The use of an autoencoder structure (encoder-decoder) helps the model learn the underlying "grammar" of neural architectures even without accuracy labels. This allows the search to be performed using gradient ascent in a continuous embedding space, effectively "walking" toward better designs.

Experimental Results

NASBench-101 Efficiency

On the NASBench-101 benchmark, SemiNAS demonstrates significant efficiency gains. It achieves comparable accuracy to gradient-based baselines while requiring only 1/7th of the architecture-accuracy queries. As shown in the results, SemiNAS reaches a 94.02% test accuracy, outperforming all baselines under the same computational budget.

Comparison of SemiNAS against baselines on NASBench-101, highlighting efficiency and accuracy gains.

ImageNet Performance

When scaled to the ImageNet dataset under a 600M FLOPS constraint, SemiNAS discovered an architecture that achieved a 23.5% top-1 error rate. This search was completed in just 4 GPU-days, showcasing the method's ability to find state-of-the-art models with limited resources.

Performance comparison on ImageNet. SemiNAS achieves the lowest Top-1 error rate among compared methods.

Applications in Speech Synthesis

SemiNAS was also applied to Text-to-Speech (TTS) on the LJSpeech dataset. Because human evaluation is too slow for NAS, the authors utilized the Diagonal Focus Rate (DFR)—a metric measuring how well a model's attention aligns diagonally between text and speech—as an automated proxy. In low-resource settings, the discovered architecture achieved a 97% intelligibility rate, a 9% improvement over human-designed Transformer-based TTS models.

Conclusion

SemiNAS significantly reduces the number of architectures that need to be trained by leveraging unlabeled data through pseudo-labeling. While it still relies on an initial "seed" of evaluated architectures, it demonstrates that semi-supervised learning can effectively break the evaluator bottleneck in NAS. Future work may involve more advanced semi-supervised techniques or applying this framework to tasks like object detection and natural language understanding.

Summary updated

Abstract

Neural architecture search (NAS) relies on a good controller to generate better architectures or predict the accuracy of given architectures. However, training the controller requires both abundant and high-quality pairs of architectures and their accuracy, while it is costly to evaluate an architecture and obtain its accuracy. In this paper, we propose SemiNAS, a semi-supervised NAS approach that leverages numerous unlabeled architectures (without evaluation and thus nearly no cost). Specifically, SemiNAS 1) trains an initial accuracy predictor with a small set of architecture-accuracy data pairs; 2) uses the trained accuracy predictor to predict the accuracy of large amount of architectures (without evaluation); and 3) adds the generated data pairs to the original data to further improve the predictor. The trained accuracy predictor can be applied to various NAS algorithms by predicting the accuracy of candidate architectures for them. SemiNAS has two advantages: 1) It reduces the computational cost under the same accuracy guarantee. On NASBench-101 benchmark dataset, it achieves comparable accuracy with gradient-based method while using only 1/7 architecture-accuracy pairs. 2) It achieves higher accuracy under the same computational cost. It achieves 94.02% test accuracy on NASBench-101, outperforming all the baselines when using the same number of architectures. On ImageNet, it achieves 23.5% top-1 error rate (under 600M FLOPS constraint) using 4 GPU-days for search. We further apply it to LJSpeech text to speech task and it achieves 97% intelligibility rate in the low-resource setting and 15% test error rate in the robustness setting, with 9%, 7% improvements over the baseline respectively.
BibTeX
@inproceedings{luo2020semi,
  author = {Renqian Luo and Xu Tan and Rui Wang and Tao Qin and Enhong Chen and Tie-Yan Liu},
  title = {{Semi-Supervised Neural Architecture Search}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=DmduyR4M9c},
}
Figure