Abstract: Neural Architecture Search methods generate large amounts of candidate architectures that need training to assess their performance and find an optimal architecture. To minimize the search time we use different performance estimation strategies. The effectiveness of such strategies varies in terms of accuracy and fit and query time. We propose Embedding proximity score (EmProx). EmProx builds a meta-model that maps candidate architectures to a continuous embedding space using an encoder-decoder framework. The performance of candidates is then estimated using weighted kNN based on the embedding vectors of architectures of which the performance is known. Performance estimations of this method are comparable to similar predictors in terms of accuracy while being nearly nine times faster to train compared to similar methods. Benchmarking against other performance estimation strategies currently used shows similar or better accuracy, while being five up to eighty times faster. Code is made publicly available on GitHub.
Introduction
Neural Architecture Search (NAS) aims to automate the design of deep learning models, but the vastness of the search space makes training every candidate architecture prohibitively expensive. To mitigate this, researchers employ performance predictors—meta-models that estimate a candidate's accuracy without full training. However, these predictors themselves, often based on complex Multi-Layer Perceptrons (MLPs) or Graph Convolutional Networks (GCNs), can become a computational bottleneck due to the time required to train (or "fit") them. This paper introduces EmProx (Embedding Proximity Score), a method designed to be exceptionally fast to train and query without sacrificing the predictive accuracy needed to guide the search process.
Methodology
EmProx builds upon the foundational framework of Neural Architecture Optimization (NAO) but introduces a critical simplification in the prediction phase. The process follows three main stages:
- Architecture Encoding: Architectures are represented as Directed Acyclic Graphs (DAGs), which are then converted into adjacency matrices and flattened into sequences for processing.
- Learning the Embedding: An Encoder-Decoder framework, utilizing LSTMs with a hidden dimension of 128, maps these sequences into a continuous 128-dimensional embedding space. The model is trained via reconstruction loss, ensuring the latent space captures the essential structural features of the architectures.
- Proximity-Based Prediction: Unlike traditional methods that train a secondary regressor, EmProx utilizes a weighted k-Nearest Neighbors (kNN) approach within the learned embedding space.
By identifying the $k$ closest architectures with known performance, EmProx estimates a new candidate's accuracy using an inverse distance-weighted average. This approach makes the prediction step nearly instantaneous and removes the need for complex regression training.
Experimental Results
The researchers evaluated EmProx using the NASLib framework on benchmarks like NAS-Bench-201 (CIFAR-10 and CIFAR-100) and the DARTS search space. The results highlight a significant efficiency advantage:
- Efficiency: EmProx was approximately 9 times faster to train than the original NAO method and between 20 to 80 times faster than other common predictors like SemiNAS or BANANAS.
- Accuracy: On CIFAR-10, EmProx achieved a Spearman correlation of ~0.73, outperforming NAO (~0.71) and remaining competitive with more computationally intensive methods like SemiNAS (~0.76).
| Method | Fit Time (s) | Spearman Correlation (CIFAR-10) |
|---|---|---|
| EmProx (k=60) | ~7.2s | 0.733 |
| NAO | ~54.1s | 0.713 |
| SemiNAS | ~152.2s | 0.767 |
| BANANAS | ~507.3s | 0.416 |
Key Insights
A central finding of this work is that a well-structured embedding space naturally encodes performance information. Using t-SNE to visualize the embeddings, the authors observed that architectures with similar performance characteristics naturally cluster together.
Specifically, poorly performing architectures (represented by red clusters) are spatially distinct, confirming that the encoder captures performance-relevant structural features even without being explicitly trained on accuracy data. This suggests that for NAS, the ability to correctly rank architectures is more important than absolute prediction accuracy, a task for which proximity-based methods are well-suited.
Conclusion and Future Work
EmProx demonstrates that high-performance NAS prediction can be achieved with minimal computational overhead. While the method shows sensitivity to hyperparameters like $k$, its speed makes it a compelling alternative for large-scale architecture searches. Future work involves integrating EmProx into end-to-end NAS pipelines to measure total wall-clock time savings and exploring the use of the embedding space for more efficient candidate sampling during the search process.
Summary updated