TL;DR
SemiNAS introduces a semi-supervised approach to Neural Architecture Search (NAS) that leverages a vast pool of unlabeled architectures to train an accuracy predictor. By using pseudo-labeling and an encoder-decoder framework, it achieves state-of-the-art results while requiring as little as 1/7th the computational cost of traditional gradient-based NAS methods.
What Problem Does This Paper Address?
Neural Architecture Search (NAS) aims to automate the design of deep learning models, but it faces a massive "evaluator bottleneck." To find a good architecture, a "controller" needs to know which designs work best. However, determining the accuracy of a single architecture typically requires training it from scratch, which is computationally ruinous (often costing hundreds or thousands of GPU days).
While "one-shot" NAS methods use weight-sharing to speed things up, they often suffer from poor ranking quality—the evaluator might think Architecture A is better than B simply because of how the shared weights were initialized, leading to sub-optimal final models. SemiNAS tackles this by asking: Can we use the millions of architectures we haven't tested to help us better understand the few we have?
Methods
The authors propose SemiNAS, which integrates semi-supervised learning into the NAS pipeline. It builds upon the Neural Architecture Optimization (NAO) framework and follows a three-step cycle:
- Initial Training: Train an accuracy predictor (an encoder-predictor-decoder model) using a small set of "labeled" data—architectures that have actually been trained and evaluated.
- Pseudo-Labeling: Use this initial predictor to estimate the accuracy of a much larger set of "unlabeled" architectures (which are "free" to generate but haven't been trained).
- Semi-Supervised Refinement: Retrain the predictor using both the small ground-truth dataset and the large pseudo-labeled dataset.
The architecture uses an LSTM encoder to map discrete graph-based architectures into a continuous embedding space, a Predictor MLP to guess the accuracy, and an LSTM decoder to reconstruct the architecture. This autoencoder structure helps the model learn the underlying "grammar" of neural architectures even without accuracy labels.
Datasets
The researchers evaluated SemiNAS across three distinct domains:
- NASBench-101: A standard benchmark containing ~423k architectures with pre-computed accuracies, used to test the efficiency of the search algorithm.
- ImageNet: The gold standard for large-scale image classification.
- LJSpeech: A public domain speech dataset used for the Text-to-Speech (TTS) task.
Key Insights and Ideas
- Unlabeled Data is "Free": While training a model is expensive, generating a random valid architecture string is nearly instantaneous. SemiNAS exploits this asymmetry.
- The Power of Latent Representations: By using an encoder-decoder (autoencoder) setup, the model learns a meaningful continuous space for architectures. This allows the search to be performed using gradient ascent—literally "walking" toward better designs in the embedding space.
- NAS for TTS: This is one of the first papers to apply NAS to Text-to-Speech. Because human evaluation (the standard for TTS) is too slow for NAS, the authors introduced the Diagonal Focus Rate (DFR). This metric measures how well the model's attention aligns diagonally between text and speech, serving as an automated proxy for speech quality.
Key Results
- Efficiency: On NASBench-101, SemiNAS achieved comparable accuracy to gradient-based baselines while using only 1/7th of the architecture-accuracy pairs (queries).
- Performance: It reached a 94.02% test accuracy on NASBench-101, outperforming all baselines when given the same computational budget.
- ImageNet: Under a 600M FLOPS constraint, it achieved a 23.5% top-1 error rate using only 4 GPU-days for the search.
- Speech Synthesis: In low-resource settings, the discovered TTS architecture achieved a 97% intelligibility rate, a 9% improvement over human-designed Transformer-based TTS models.
Limitations and Future Work
While SemiNAS significantly reduces the number of architectures that need to be trained, it still relies on an initial "seed" of evaluated architectures. If the initial sample is too small or biased, the pseudo-labels might lead the predictor astray.
The authors suggest that future work could involve more advanced semi-supervised techniques (like MixMatch) or applying the framework to a wider variety of tasks beyond classification and speech, such as natural language understanding or object detection.
Summary updated