Abstract
Neural architecture search methods are able to find high performance deep learning architectures with minimal effort from an expert. However, current systems focus on specific use-cases (e.g. convolutional image classifiers and recurrent language models), making them unsuitable for general use-cases that an expert might wish to write. Hyperparameter optimization systems are general-purpose but lack the constructs needed for easy application to architecture search. In this work, we propose a formal language for encoding search spaces over general computational graphs. The language constructs allow us to write modular, composable, and reusable search space encodings and to reason about search space design. We use our language to encode search spaces from the architecture search literature. The language allows us to decouple the implementations of the search space and the search algorithm, allowing us to expose search spaces to search algorithms through a consistent interface. Our experiments show the ease with which we can experiment with different combinations of search spaces and search algorithms without having to implement each combination from scratch. We release an implementation of our language with this paper.
Introduction
Neural Architecture Search (NAS) promises to automate the design of deep learning models, yet the field is hampered by a lack of modularity. Most NAS implementations tightly couple the search space—the set of possible architectures—with the search algorithm. This "intertwining" forces researchers to rewrite entire frameworks when testing new ideas, making fair benchmarking and component reuse nearly impossible. While general hyperparameter optimization (HPO) tools exist, they typically treat architectures as flat vectors, failing to capture the complex, conditional, and hierarchical nature of computational graphs.
A Language for Modular Search Spaces
To address these limitations, we introduce a formal language designed to encode search spaces as programmable computational graphs. This approach allows experts to write "programs" that describe model construction rather than static templates.
Key Language Constructs
The language relies on several core primitives:
- Independent and Dependent Hyperparameters: These define the decision points (e.g., filter counts or kernel sizes). Dependent hyperparameters allow for conditional logic where choices depend on previous assignments.
- Basic and Substitution Modules: Basic modules wrap standard operations. Substitution modules are the "lazy" core of the language; they delay sub-graph creation until specific hyperparameters are assigned, enabling the representation of complex, branching search spaces.
- Auxiliary Functions: These higher-order functions facilitate the composition of small, reusable search spaces into larger structures, such as repeating a specific cell pattern.
By decoupling the search space from the algorithm, any space defined in this language can be explored by any compatible search algorithm through a unified interface. The system then compiles the chosen hyperparameters into a runnable TensorFlow or PyTorch graph.
Experimental Evaluation
We evaluated the framework by reimplementing prominent search spaces, including Nasbench-101, Nasnet, and others, testing them on CIFAR-10 and Penn Treebank (PTB) datasets.
The Primacy of Search Space Design
A key insight from our experiments is that the design of the search space often exerts a greater influence on final performance than the choice of the search algorithm itself. When testing four distinct algorithms (Random Search, MCTS, SMBO, and Regularized Evolution) on the same "Genetic" search space, performance remained remarkably consistent, with test accuracies clustered between 91.32% and 91.93%.
In contrast, switching the search space while keeping the algorithm constant led to significant performance swings. For instance, the "Nasbench" space reached 94.59% accuracy using the same methodology that yielded ~90% in the "Genetic" space. This underscores the value of a modular language that allows researchers to focus on search space engineering.
Conclusion
Our proposed language provides a programmable foundation for NAS, enabling the creation of complex, reusable, and algorithm-agnostic search spaces. By facilitating the "mix and match" of components, we demonstrate that search space design is a critical, yet often overlooked, factor in NAS performance. Future work will focus on integrating efficiency-enhancing techniques like weight sharing into this modular framework and extending the approach to broader machine learning pipelines.
The implementation is available in the open-source deep_architect library.
Summary updated