A Study of the Learning Progress in Neural Architecture Search Techniques

Prabhant Singh, Tobias Jacobs, Sebastien Nicolas, Mischa SchmidtarXiv 2019 · OpenReview

TL;DR

This paper investigates the learning progress of Efficient Neural Architecture Search (ENAS), a popular method for automatically designing neural networks. Surprisingly, they found that the performance of architectures generated by ENAS doesn't improve significantly during the controller's training, suggesting that the search space design, rather than the controller's learning, is the primary driver of good performance. This implies that simpler, one-shot approaches might be just as effective and much faster.

What Problem Does This Paper Address?

Neural Architecture Search (NAS) aims to automate the discovery of optimal neural network architectures for specific tasks, a process that has historically been computationally expensive. Efficient Neural Architecture Search (ENAS) emerged as a promising technique, significantly reducing search time by sharing weights across candidate architectures and training a controller to generate them. However, the authors were motivated by the desire to further accelerate this process and questioned when during the ENAS training the best architectures are actually found. They sought to understand if the controller's learning truly contributes to finding better architectures over time, or if good architectures are discovered early and the subsequent training is less impactful.

Methods

The core of this study involves re-evaluating the ENAS procedure. The authors implemented ENAS and systematically sampled architectures generated by the controller at various stages of its training. Crucially, each sampled architecture was then fully retrained from scratch on the target dataset to obtain its "exact performance" (test accuracy). This allowed them to track how the performance of these independently trained architectures evolved as the controller's training progressed. They compared architectures generated early in the controller's training with those generated much later, after hundreds of training epochs.

Datasets

  • CIFAR-10: A widely used benchmark dataset for image classification.
  • CIFAR-100: A more challenging version of CIFAR-10 with a larger number of classes.

The paper also mentions that ENAS was originally designed for language processing tasks, but their experiments focused on CNN architectures for image classification.

Key Insights and Ideas

  • "Flat" Learning Curves: The most striking finding is that the test accuracy of architectures generated by ENAS shows virtually no improvement as the controller trains. This suggests that the controller isn't progressively learning to generate better architectures over time.
  • Search Space Dominance: The authors hypothesize that the high performance achieved by ENAS is primarily due to the carefully designed search space, which likely contains many good architectures, rather than the controller's ability to learn and optimize.
  • One-Shot as a Viable Alternative: Given the lack of learning progress, the paper argues that skipping the ENAS search process and directly using architectures found early (or even randomly sampled from a good search space) could yield comparable results with significant time savings. This positions "one-shot" methods, where an architecture is evaluated only once, as a strong alternative.
  • Validation Accuracy vs. Test Accuracy Disconnect: Figure 1 illustrates that the validation accuracy (using shared weights) of a generated architecture does not reliably correlate with its final test accuracy (after retraining). This implies that the feedback signal used by the ENAS controller might not be a good predictor of true performance.

Key Results

  • The paper reports that the test accuracies of architectures generated by ENAS at different controller training epochs are remarkably similar. For instance, on CIFAR-10 (macro search space), initial accuracies were around 95.71-96.69%, and final accuracies were around 95.38-95.81%. Similar trends were observed across different datasets and search spaces.
  • The improvement in test accuracy when comparing architectures found early versus late in the search process was consistently below 1% across most configurations, even on the more challenging CIFAR-100 dataset.
  • This suggests that the computationally intensive search process (taking several GPU-hours) can often be skipped without sacrificing accuracy.

Limitations and Future Work

  • Limited Runs: The authors acknowledge that their results are based on a specific set of experiment runs and suggest that future work should include more extensive reproduction of these results.
  • Focus on CNNs: The current study focused on CNN architectures. Future work should extend these findings to recurrent neural networks (RNNs) and other architectures.
  • Understanding Validation Accuracy: A more systematic study on the prediction quality of validation accuracy (when using tentative shared weights) is proposed to better understand the effectiveness of speedup techniques like ENAS.
  • Generalizability: The methodology of studying learning progress is suggested to be applicable to evaluating other NAS techniques as well.

Summary updated

Abstract

In neural architecture search, the structure of the neural network to best model a given dataset is determined by an automated search process. Efficient Neural Architecture Search (ENAS), proposed by Pham et al. (2018), has recently received considerable attention due to its ability to find excellent architectures within a comparably short search time. In this work, which is motivated by the quest to further improve the learning speed of architecture search, we evaluate the learning progress of the controller which generates the architectures in ENAS. We measure the progress by comparing the architectures generated by it at different controller training epochs, where architectures are evaluated after having re-trained them from scratch. As a surprising result, we find that the learning curves are completely flat, i.e., there is no observable progress of the controller in terms of the performance of its generated architectures. This observation is consistent across the CIFAR-10 and CIFAR-100 datasets and two different search spaces. We conclude that the high quality of the models generated by ENAS is a result of the search space design rather than the controller training, and our results indicate that one-shot architecture design is an efficient alternative to architecture search by ENAS.
BibTeX
@misc{singh2019study,
  author = {Prabhant Singh and Tobias Jacobs and Sebastien Nicolas and Mischa Schmidt},
  title = {{A Study of the Learning Progress in Neural Architecture Search Techniques}},
  year = {2019},
  eprint = {1906.07590},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.07590},
}
Figure