Search Tree Pruning for Progressive Neural Architecture Search (Student Abstract)

Deanna Flynn, P. Michael Furlong, Brian ColtinAAAI 2020 · DOI

Introduction

Neural Architecture Search (NAS) has significantly advanced the automated design of deep learning models. However, state-of-the-art NAS techniques typically require immense computational resources, often involving clusters of high-end GPUs to explore complex "cell-based" search spaces. This creates a barrier for researchers and developers with limited hardware. This paper addresses this gap by proposing a progressive NAS algorithm designed to find efficient, low-parameter models that can be trained on consumer-grade hardware without sacrificing significant accuracy.

Methodology

The proposed approach utilizes a progressive search algorithm derived from Levin search. The algorithm explores the space of possible architectures by building a search tree using a Depth First Search (DFS) strategy. Starting from simple configurations, the system incrementally adds layers to "leaf-node" networks until a predefined maximum depth is reached.

Transition Graph and Constraints

To ensure the search space remains manageable and produces valid architectures, the authors introduce a "transition graph." This graph serves as the primary point of human intervention, defining the permissible sequences of layers—such as Convolution, Dense, Dropout, Flatten, and Max Pooling. By constraining the search to sequences supported by existing literature, the algorithm avoids exploring nonsensical structures.

The human-defined transition graph used to constrain the architecture search space.

Inheritance and Greedy Pruning

Efficiency is further enhanced through two key mechanisms:

  1. Inheritance: Child nodes in the search tree inherit weights and hyperparameters from their parent nodes, allowing the search to build upon previously learned features rather than starting from scratch.
  2. Greedy Pruning: To accelerate the search, the algorithm employs a pruning strategy. If a child network fails to outperform its parent after a set number of training epochs, that entire branch of the search tree is discarded. While this "greedy" approach might overlook architectures that require more depth to mature, it drastically reduces the total search time.

Experimental Setup

The algorithm was evaluated on the Fashion-MNIST dataset, which consists of 60,000 grayscale images across 10 fashion categories. The authors utilized the unprocessed version of the dataset to rigorously test the raw feature-extraction capabilities of the generated architectures.

Results

The search process explored 61 different architectures over four days, running entirely on a single Intel i7 8th Generation CPU. The resulting model achieved a remarkable balance between size and performance.

Comparison of model size, training time, and hardware requirements.

As shown in the table above, the discovered model contains only 98,000 parameters—a fraction of the size of industry-standard models like GoogleNet (4M) or AlexNet (~60M). Despite this reduction in complexity, the model achieved an accuracy of 91.9% on Fashion-MNIST, trailing GoogleNet by less than 2%. Crucially, while standard models often require days of training on multiple high-end GPUs (e.g., Nvidia Titan Black), the proposed model was found and trained using only a consumer-grade CPU.

Discussion and Future Work

The success of this method underscores the power of "simplicity first" design. By starting with minimal models and using a transition graph as a prior, the algorithm naturally discovers efficient structures before reaching more complex, "bloated" alternatives.

However, several areas for improvement remain. The current greedy pruning rule may be too aggressive, potentially missing optimal networks that exhibit non-monotonic performance gains. Additionally, hyperparameter optimization currently accounts for over 50% of the total runtime. Future iterations could integrate Bayesian optimization to streamline this process. Finally, while the transition graph is a powerful tool, its reliance on human design introduces bias; automating the evolution of this graph could lead to even more innovative architectural discoveries.

Summary updated

BibTeX
@inproceedings{flynn2020search,
  author = {Deanna Flynn and P. Michael Furlong and Brian Coltin},
  title = {{Search Tree Pruning for Progressive Neural Architecture Search (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7163},
}
Figure