Abstract
Neural Architecture Search (NAS) aims to automate the design of deep learning models, yet current methodologies are often over-tuned to a small set of "celebrity" datasets like CIFAR-10. This paper investigates the generalization capabilities of NAS by introducing eight diverse, previously unseen datasets. The results reveal that state-of-the-art search algorithms frequently struggle with novel problems, sometimes failing to outperform simple random search baselines. This work highlights the need for more robust evaluation protocols to achieve true democratization in deep learning.
Introduction: The Over-Engineering Trap
The ultimate goal of Deep Learning is democratization—enabling users to find high-performing models for new tasks without requiring a human expert to hand-craft every architecture. Neural Architecture Search (NAS) was proposed as the solution to this problem. However, the NAS community has largely focused its research on a narrow set of benchmarks, specifically CIFAR-10, CIFAR-100, and ImageNet.
Because these datasets are so well-understood, researchers may inadvertently design search spaces and strategies that are biased toward them. This paper addresses the unseen data challenge: can NAS frameworks actually find superior models for problems they were not specifically engineered to solve?
Methodology and the Unseen Benchmarks
To evaluate the state of the field, the authors organized a competition requiring participants to develop NAS methods for testing on entirely novel datasets. The evaluation compared three categories of models:
- Standard CNNs: Established architectures like ResNet-18, AlexNet, VGG16, and DenseNet.
- NAS Frameworks: Popular automated methods including PC-DARTS, DrNAS, and Bonsai-Net.
- Random Search: A baseline that selects architectures randomly from a predefined search space.
The study introduced eight new benchmarks categorized into two types. Type-1 problems involve logic that a human could solve with a simple program, such as the AddNIST and MultNIST datasets where digits are hidden in RGB channels and the label is their sum or product.
Type-2 problems are more complex and often non-intuitive for humans without specialized tools. These include:
- Language: Identifying languages from a 24x24 grid encoding of word frequencies.
- CIFARTile: Counting unique classes in a 2x2 grid of CIFAR images.
- Gutenberg: Identifying authors based on spatial patterns of letter sequences.
- Isabella: Classifying musical eras from audio spectrograms.
- GeoClassing: Country identification from satellite imagery.
- Chesseract: Predicting chess game winners from a 12-channel board encoding.
Key Insights: The "Expert Shift"
The experiments yielded several critical insights into the current state of NAS:
The Persistence of the "No Free Lunch" Theorem
No single architecture dominated across all tasks. Interestingly, older models like AlexNet performed surprisingly well on specific tasks like AddNIST because their inherent structure happened to align with the spatial layout of those problems. Conversely, DenseNet and ResNeXt proved robust for memory-intensive tasks and those requiring the detection of subtle patterns.
NAS Overfitting and Random Search
Many NAS methods are so specialized for CIFAR-style images that they fail to generalize to the new benchmarks. In several instances, random search outperformed sophisticated gradient-based NAS methods like PC-DARTS and DrNAS within the DARTS search space. This suggests that the complexity of modern NAS algorithms may not always translate to better performance on novel data.
The Search Space Bottleneck
NAS has not removed the need for human experts; it has merely shifted their responsibilities. Instead of designing the network, experts now design the search space. If the search space lacks the necessary building blocks for a specific task—such as the 12-channel input required for Chesseract—the NAS algorithm cannot succeed.
Experimental Results
While NAS found the best model for 7 out of the 8 datasets, the margin of victory over standard CNNs was often slim.
Notable performance highlights include:
- CIFARTile: PC-DARTS achieved a significant breakthrough with 92.28% accuracy, compared to just 45.56% for a standard ResNet-18.
- Language: A standard ResNeXt model hit 93.97%, outperforming all automated NAS attempts.
- Chesseract: This proved to be the most difficult task, with the top competition submission reaching only 62.98% (where random chance is 33.3%).
Conclusion and Future Directions
The study concludes that the NAS community must move away from static benchmarks. The authors advocate for a "rolling" benchmark system that constantly introduces new, diverse datasets. This would force researchers to focus on true architectural generalization rather than optimizing for a handful of well-known datasets. Future work should also explore more flexible search spaces that can adapt to non-standard data formats beyond traditional image grids.
Summary updated