TL;DR
The field of Neural Architecture Search (NAS) is experiencing explosive growth, but its empirical evaluation standards are currently lacking. This leads to results that are often irreproducible and difficult to compare. To address this, the authors introduce a comprehensive 14-point checklist of best practices for NAS research, focusing on code release, fair comparisons, and detailed reporting to ensure scientifically rigorous progress.
The Reproducibility Crisis in NAS
NAS aims to automate the design of neural network architectures, a task traditionally requiring significant human expertise. While the volume of NAS research is growing exponentially, the scientific rigor of its evaluations has not kept pace with other machine learning domains.
Researchers frequently compare "apples to oranges" by using disparate training pipelines, omitting hyperparameter tuning details, or evaluating on inconsistent search spaces. This lack of standardization makes it difficult to determine if a new NAS method offers a genuine algorithmic improvement or simply benefits from superior data augmentation or engineering tricks. The absence of open-source code and comparisons against simple baselines, such as random search, has fueled skepticism regarding the true efficacy of many proposed methods.
A Framework for Rigorous NAS Research
Instead of proposing a new algorithm, this paper proposes a methodological framework for conducting and reporting NAS research. The authors outline 14 specific best practices divided into three main categories:
- Releasing Code: Researchers should release the complete training pipeline, not just the NAS algorithm itself. The training setup often has a greater impact on final performance than the architecture. The authors advocate for releasing prototype code immediately to facilitate community verification.
- Standardized Comparisons: Evaluations should utilize established NAS benchmarks, include thorough ablation studies, and track performance relative to compute resources. Crucially, every new method must be compared against random search and random sampling baselines.
- Transparent Reporting: Documentation must include hyperparameter optimization (HPO) steps, end-to-end compute costs (including validation), and specific hardware/software environments.
The Role of Benchmarks
The paper critiques the field's over-reliance on a few datasets like CIFAR-10, which risks "meta-overfitting." Instead, the authors champion the use of Tabular and Surrogate NAS Benchmarks.
These benchmarks (e.g., NAS-Bench-101, 201, 301) provide pre-computed performance data for thousands or even billions of architectures. They allow researchers to conduct rigorous, multi-seed experiments in seconds, democratizing NAS research for those without massive GPU clusters.
Key Insights
- Pipeline vs. Architecture: Modern training tricks (e.g., CutOut, MixUp) often contribute more to performance gains than the architecture search itself. In some cases, training improvements yielded a 3% accuracy boost, while the search algorithm contributed less than 1%.
- Random Search as a Baseline: Many complex NAS methods fail to outperform simple random search when evaluated fairly. Without this baseline, the utility of a search algorithm remains unproven.
- HPO Costs: If an algorithm requires extensive manual tuning for new datasets, that time must be factored into the total compute cost. NAS should not simply replace manual architecture engineering with manual hyperparameter engineering.
Future Directions
While adhering to all best practices can be computationally demanding, the authors suggest that the community focus on two areas to push the field forward:
- Diverse Benchmarks: Expanding benchmarks to include tasks like semantic segmentation, GANs, and machine translation to prevent domain-specific overfitting.
- Unified Libraries: Developing open-source libraries (similar to RLlib for reinforcement learning) to provide a common interface for comparing NAS methods without confounding factors.
By adopting the NAS Best Practices Checklist, the community can move toward a more transparent and scientifically sound future.
Summary updated