Introduction
Neural Architecture Search (NAS) has significantly advanced the automated design of deep learning models. However, state-of-the-art NAS techniques typically require immense computational resources, often involving clusters of high-end GPUs to explore complex "cell-based" search spaces. This creates a barrier for researchers and developers with limited hardware. This paper addresses this gap by proposing a progressive NAS algorithm designed to find efficient, low-parameter models that can be trained on consumer-grade hardware without sacrificing significant accuracy.
Methodology
The proposed approach utilizes a progressive search algorithm derived from Levin search. The algorithm explores the space of possible architectures by building a search tree using a Depth First Search (DFS) strategy. Starting from simple configurations, the system incrementally adds layers to "leaf-node" networks until a predefined maximum depth is reached.
Transition Graph and Constraints
To ensure the search space remains manageable and produces valid architectures, the authors introduce a "transition graph." This graph serves as the primary point of human intervention, defining the permissible sequences of layers—such as Convolution, Dense, Dropout, Flatten, and Max Pooling. By constraining the search to sequences supported by existing literature, the algorithm avoids exploring nonsensical structures.
Inheritance and Greedy Pruning
Efficiency is further enhanced through two key mechanisms:
- Inheritance: Child nodes in the search tree inherit weights and hyperparameters from their parent nodes, allowing the search to build upon previously learned features rather than starting from scratch.
- Greedy Pruning: To accelerate the search, the algorithm employs a pruning strategy. If a child network fails to outperform its parent after a set number of training epochs, that entire branch of the search tree is discarded. While this "greedy" approach might overlook architectures that require more depth to mature, it drastically reduces the total search time.
Experimental Setup
The algorithm was evaluated on the Fashion-MNIST dataset, which consists of 60,000 grayscale images across 10 fashion categories. The authors utilized the unprocessed version of the dataset to rigorously test the raw feature-extraction capabilities of the generated architectures.
Results
The search process explored 61 different architectures over four days, running entirely on a single Intel i7 8th Generation CPU. The resulting model achieved a remarkable balance between size and performance.
As shown in the table above, the discovered model contains only 98,000 parameters—a fraction of the size of industry-standard models like GoogleNet (4M) or AlexNet (~60M). Despite this reduction in complexity, the model achieved an accuracy of 91.9% on Fashion-MNIST, trailing GoogleNet by less than 2%. Crucially, while standard models often require days of training on multiple high-end GPUs (e.g., Nvidia Titan Black), the proposed model was found and trained using only a consumer-grade CPU.
Discussion and Future Work
The success of this method underscores the power of "simplicity first" design. By starting with minimal models and using a transition graph as a prior, the algorithm naturally discovers efficient structures before reaching more complex, "bloated" alternatives.
However, several areas for improvement remain. The current greedy pruning rule may be too aggressive, potentially missing optimal networks that exhibit non-monotonic performance gains. Additionally, hyperparameter optimization currently accounts for over 50% of the total runtime. Future iterations could integrate Bayesian optimization to streamline this process. Finally, while the transition graph is a powerful tool, its reliance on human design introduces bias; automating the evolution of this graph could lead to even more innovative architectural discoveries.
Summary updated