Large Scale Neural Architecture Search with Polyharmonic Splines

Ulrich Finkler, Michele Merler, Rameswar Panda, Mayoore S. Jaiswal, Hui Wu, Kandan Ramakrishnan, Chun-Fu Chen, Minsik Cho, David S. Kung, Rogério Feris, Bishwaranjan BhattacharjeearXiv 2020 · OpenReview

Abstract

Neural Architecture Search (NAS) has revolutionized the design of deep neural networks, yet its application to large-scale, imbalanced datasets remains a significant computational challenge. Traditional NAS methods often rely on small proxy datasets, transferring discovered architectures to larger tasks by replicating or stacking cells—a process that frequently fails to capture the complexity of real-world data. This paper introduces a novel NAS method based on polyharmonic splines that enables direct search on large-scale datasets. By modeling the architecture search space as a continuous, smooth landscape, the proposed method achieves a state-of-the-art top-1 accuracy of 40.03% on the ImageNet22K benchmark, representing a 3.13% absolute improvement over previous benchmarks.

Introduction

Designing optimal neural architectures manually is a labor-intensive process. While NAS automates this, the computational cost of evaluating thousands of candidate models is prohibitive for massive datasets like ImageNet22K, which contains 14 million images across 21,841 categories. To circumvent this, researchers typically perform searches on small, balanced "proxy" sets (e.g., CIFAR-10) and scale the results. However, architectures optimized for small-scale data often struggle with the "long tail" distribution and high-dimensional category spaces of large-scale benchmarks.

This research addresses the scalability bottleneck by treating NAS as a high-dimensional interpolation problem. By utilizing polyharmonic splines, the authors demonstrate that it is possible to navigate a search space of trillions of combinations with only a few dozen full model evaluations.

Methodology: Polyharmonic Spline NAS

The core innovation of this work is the application of Polyharmonic Splines to model the relationship between architecture parameters and model performance.

The Accuracy-Hypersurface

The authors hypothesize that because standard neural network components—such as ReLU activations, Convolutions, and Batch Normalization—are essentially piecewise linear, the resulting accuracy-hypersurface of the search space is likely smooth and well-behaved. This smoothness allows the search space to be treated as a continuous landscape that can be interpolated using radial basis functions.

The Search Algorithm

Unlike Reinforcement Learning (RL) or Gradient-based NAS, which require massive iteration counts, the spline-based approach follows a highly efficient iterative cycle:

  1. Initial Sampling: Evaluate a small set of initial architectures ($2d + 3$ points, where $d$ is the number of dimensions in the search space).
  2. Spline Fitting: Fit a polyharmonic spline to the observed accuracy data.
  3. Peak Finding: Use a Monte Carlo approach to identify the predicted maximum accuracy on the interpolated spline.
  4. Verification and Refinement: Train the model at the predicted peak, add the actual result to the dataset, and refine the spline model.

This approach ensures that the number of evaluations scales with the number of architectural dimensions (e.g., width, depth) rather than the total number of possible configurations.

Experimental Setup and Datasets

The method was primarily evaluated on ImageNet22K, a dataset characterized by its massive scale and high class imbalance. For smaller-scale verification and convergence testing, ImageNet1K was also utilized.

One significant advantage of this method is its memory efficiency. Unlike "Supernet" approaches that attempt to store multiple architecture variants in GPU memory simultaneously, the spline method evaluates one model at a time. This allows for the inclusion of very large models in the search space that would typically exceed the memory limits of a Supernet.

Key Insights and Results

Performance on ImageNet22K

The search focused on macro-parameters within the ResNet and Big-Little Net (BLResNext) families. The discovered architecture achieved a 40.03% top-1 accuracy on ImageNet22K. This marks a 3.13% absolute improvement over the previous state-of-the-art for models trained with similar global batch sizes.

Efficiency Gains

The computational efficiency of the spline method is stark. The search required approximately 30,000 GPU hours on the Summit Supercomputer. While substantial, the authors estimate that a traditional RL-based NAS would have required over 13 million GPU hours to explore the same search space of 3 trillion possible combinations.

Architectural Observations

The study revealed that for large-scale datasets with many categories, increasing the width of later layers is the dominant factor in performance. The authors also identified a "choke point" phenomenon: if leading layers reduce the dimensions of the vector space too aggressively, the final accuracy is negatively impacted regardless of subsequent layer complexity.

Conclusion and Future Work

This work demonstrates that direct NAS on large-scale, imbalanced datasets is not only tractable but superior to proxy-based methods. By leveraging the mathematical properties of polyharmonic splines, the researchers bypassed the traditional computational barriers of architecture search.

Future research may expand this framework to include "micro" search (discovering new layer types) and the simultaneous optimization of training hyperparameters, such as learning rate schedules and data augmentation strategies, within the same spline-based interpolation space.

Summary updated

Abstract

Neural Architecture Search (NAS) is a powerful tool to automatically design deep neural networks for many tasks, including image classification. Due to the significant computational burden of the search phase, most NAS methods have focused so far on small, balanced datasets. All attempts at conducting NAS at large scale have employed small proxy sets, and then transferred the learned architectures to larger datasets by replicating or stacking the searched cells. We propose a NAS method based on polyharmonic splines that can perform search directly on large scale, imbalanced target datasets. We demonstrate the effectiveness of our method on the ImageNet22K benchmark[16], which contains 14 million images distributed in a highly imbalanced manner over 21,841 categories. By exploring the search space of the ResNet [23] and Big-Little Net ResNext [11] architectures directly on ImageNet22K, our polyharmonic splines NAS method designed a model which achieved a top-1 accuracy of 40.03% on ImageNet22K, an absolute improvement of 3.13% over the state of the art with similar global batch size [15].
BibTeX
@misc{finkler2020large,
  author = {Ulrich Finkler and Michele Merler and Rameswar Panda and Mayoore S. Jaiswal and Hui Wu and Kandan Ramakrishnan and Chun-Fu Chen and Minsik Cho and David S. Kung and Rogério Feris and Bishwaranjan Bhattacharjee},
  title = {{Large Scale Neural Architecture Search with Polyharmonic Splines}},
  year = {2020},
  eprint = {2011.10608},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.10608},
}
Figure