Abstract
Neural Architecture Search (NAS) has revolutionized the design of deep neural networks, yet its application to large-scale, imbalanced datasets remains a significant computational challenge. Traditional NAS methods often rely on small proxy datasets, transferring discovered architectures to larger tasks by replicating or stacking cells—a process that frequently fails to capture the complexity of real-world data. This paper introduces a novel NAS method based on polyharmonic splines that enables direct search on large-scale datasets. By modeling the architecture search space as a continuous, smooth landscape, the proposed method achieves a state-of-the-art top-1 accuracy of 40.03% on the ImageNet22K benchmark, representing a 3.13% absolute improvement over previous benchmarks.
Introduction
Designing optimal neural architectures manually is a labor-intensive process. While NAS automates this, the computational cost of evaluating thousands of candidate models is prohibitive for massive datasets like ImageNet22K, which contains 14 million images across 21,841 categories. To circumvent this, researchers typically perform searches on small, balanced "proxy" sets (e.g., CIFAR-10) and scale the results. However, architectures optimized for small-scale data often struggle with the "long tail" distribution and high-dimensional category spaces of large-scale benchmarks.
This research addresses the scalability bottleneck by treating NAS as a high-dimensional interpolation problem. By utilizing polyharmonic splines, the authors demonstrate that it is possible to navigate a search space of trillions of combinations with only a few dozen full model evaluations.
Methodology: Polyharmonic Spline NAS
The core innovation of this work is the application of Polyharmonic Splines to model the relationship between architecture parameters and model performance.
The Accuracy-Hypersurface
The authors hypothesize that because standard neural network components—such as ReLU activations, Convolutions, and Batch Normalization—are essentially piecewise linear, the resulting accuracy-hypersurface of the search space is likely smooth and well-behaved. This smoothness allows the search space to be treated as a continuous landscape that can be interpolated using radial basis functions.
The Search Algorithm
Unlike Reinforcement Learning (RL) or Gradient-based NAS, which require massive iteration counts, the spline-based approach follows a highly efficient iterative cycle:
- Initial Sampling: Evaluate a small set of initial architectures ($2d + 3$ points, where $d$ is the number of dimensions in the search space).
- Spline Fitting: Fit a polyharmonic spline to the observed accuracy data.
- Peak Finding: Use a Monte Carlo approach to identify the predicted maximum accuracy on the interpolated spline.
- Verification and Refinement: Train the model at the predicted peak, add the actual result to the dataset, and refine the spline model.
This approach ensures that the number of evaluations scales with the number of architectural dimensions (e.g., width, depth) rather than the total number of possible configurations.
Experimental Setup and Datasets
The method was primarily evaluated on ImageNet22K, a dataset characterized by its massive scale and high class imbalance. For smaller-scale verification and convergence testing, ImageNet1K was also utilized.
One significant advantage of this method is its memory efficiency. Unlike "Supernet" approaches that attempt to store multiple architecture variants in GPU memory simultaneously, the spline method evaluates one model at a time. This allows for the inclusion of very large models in the search space that would typically exceed the memory limits of a Supernet.
Key Insights and Results
Performance on ImageNet22K
The search focused on macro-parameters within the ResNet and Big-Little Net (BLResNext) families. The discovered architecture achieved a 40.03% top-1 accuracy on ImageNet22K. This marks a 3.13% absolute improvement over the previous state-of-the-art for models trained with similar global batch sizes.
Efficiency Gains
The computational efficiency of the spline method is stark. The search required approximately 30,000 GPU hours on the Summit Supercomputer. While substantial, the authors estimate that a traditional RL-based NAS would have required over 13 million GPU hours to explore the same search space of 3 trillion possible combinations.
Architectural Observations
The study revealed that for large-scale datasets with many categories, increasing the width of later layers is the dominant factor in performance. The authors also identified a "choke point" phenomenon: if leading layers reduce the dimensions of the vector space too aggressively, the final accuracy is negatively impacted regardless of subsequent layer complexity.
Conclusion and Future Work
This work demonstrates that direct NAS on large-scale, imbalanced datasets is not only tractable but superior to proxy-based methods. By leveraging the mathematical properties of polyharmonic splines, the researchers bypassed the traditional computational barriers of architecture search.
Future research may expand this framework to include "micro" search (discovering new layer types) and the simultaneous optimization of training hyperparameters, such as learning rate schedules and data augmentation strategies, within the same spline-based interpolation space.
Summary updated