Are Labels Necessary for Neural Architecture Search?

Chenxi Liu, Piotr Dollár, Kaiming He, Ross B. Girshick, Alan L. Yuille, Saining Xie

Introduction

Neural Architecture Search (NAS) has revolutionized computer vision by automating the design of high-performance networks. However, traditional NAS relies heavily on the supervised learning regime, requiring massive datasets of human-annotated labels (like ImageNet) to evaluate candidate architectures during the search process. This creates a significant bottleneck for domains where labels are scarce or non-existent.

This research investigates a fundamental question: Are labels indispensable for finding good architectures, or is the "quality" of a neural network inherent to how it processes the statistics of raw images? To answer this, the authors introduce Unsupervised NAS (UnNAS), demonstrating that image statistics alone are sufficient to identify high-performing architectures.

The UnNAS Framework

In the UnNAS framework, the search phase utilizes no labels, while the evaluation phase (training the final discovered model) uses standard supervised labels to measure performance. This approach decouples the architecture search from the specific task labels.

Conceptual framework of Unsupervised NAS (UnNAS) compared to standard unsupervised learning.

The authors evaluated this through two tracks: sample-based experiments (correlating performance across 500 random architectures) and search-based experiments (modifying the DARTS algorithm). They utilized three "pretext" unsupervised tasks to guide the search:

  • Rotation Prediction: Predicting if an image was rotated 0, 90, 180, or 270 degrees.
  • Colorization: Predicting the color of a grayscale image.
  • Jigsaw Puzzles: Reassembling shuffled image patches.

Key Insights: Intrinsic Architecture Quality

The most striking finding is the high Spearman’s Rank Correlation ($\rho$) between unsupervised and supervised performance. The study found that the ranking of architectures is highly consistent ($ ho > 0.8$) regardless of whether labels are used. If an architecture is better at solving a jigsaw puzzle, it is almost certainly better at standard image classification.

Correlation between unsupervised pretext task performance and supervised ImageNet accuracy across different search spaces.

As illustrated in the correlation plots, the performance on pretext tasks serves as an excellent proxy for supervised generalization. This suggests that a "good" architecture is one that effectively captures the underlying structure of natural images; labels are just one way to reveal this latent quality.

Key Results and Performance

  • Competitive Performance: On ImageNet-1K, architectures found via UnNAS-DARTS achieved up to 75.9% top-1 accuracy, nearly matching the 76.3% achieved by supervised DARTS.
  • Unsupervised Wins in Segmentation: When searching on the Cityscapes dataset for semantic segmentation, UnNAS-DARTS (using the Jigsaw objective) achieved 74.1 mIoU, actually outperforming the supervised search result of 72.4 mIoU.
  • Scalability: The research demonstrates that searching on large, unlabeled datasets like ImageNet-22K is a viable strategy, potentially allowing AI to design itself using the vast amounts of unlabeled data available on the internet.

Conclusion

This paper shifts the paradigm of NAS from "learning from labels" to "learning from data." While search space bias remains a factor (as current spaces were designed with supervised performance in mind), the results clearly indicate that the secret to a great neural architecture is hidden in the pixels, not the annotations. UnNAS opens the door to automated model design in any domain where raw data is plentiful, even if human labels are not.

Summary updated

BibTeX
@inproceedings{liu2020are,
  author = {Chenxi Liu and Piotr Dollár and Kaiming He and Ross B. Girshick and Alan L. Yuille and Saining Xie},
  title = {{Are Labels Necessary for Neural Architecture Search?}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58548-8\_46},
}
Figure