Abstract
In this paper, we propose a neural architecture search (NAS) framework based on a similarity measure between baseline tasks and a target task. We define task similarity using the log-determinant of the Fisher Information Matrix (FIM). By quantifying the relationship between a target and a set of learned baseline tasks, the search space for the target task can be significantly reduced. This approach makes the discovery of optimal architectures tractable and efficient, requiring only 2 GPU days. Our method eliminates the need for training networks from scratch for new tasks and reduces human bias in search space initialization.
Introduction
Neural Architecture Search (NAS) represents a significant step toward fully automated machine learning, yet it remains computationally prohibitive, often requiring thousands of GPU days. Traditional NAS methods typically suffer from two limitations: they ignore knowledge from previous similar tasks (starting from zero) and rely on human-defined search spaces that may limit architectural diversity.
We introduce LD-NAS, a framework that bridges transfer learning and NAS. By identifying which previously solved tasks are most similar to a new target task, LD-NAS reuses proven architectural search spaces, drastically reducing the computational overhead while maintaining or improving performance.
Methodology
Task Similarity via Fisher Information
The core of LD-NAS is a dissimilarity measure $d[b, t]$ based on the log-determinant of the Fisher Information Matrix (FIM). The FIM captures how much information the data provides about model parameters. To ensure computational tractability, we utilize a diagonal approximation of the FIM, reducing the complexity from $n^2$ to $n$ entries.
As shown in the similarity heatmap, this distance is asymmetric; transferring knowledge from a complex task to a simpler one is often more effective than the reverse. Interestingly, structural complexity often outweighs domain similarity. For instance, Task 7 (CIFAR-10) was found to be a close match for Task 3 (MNIST), as both involve 10-class classification, despite the visual differences between natural objects and grayscale digits.
Search Space Reduction and Optimization
Once the most similar "base task" is identified, LD-NAS adopts its architectural search space. We then apply the FUSE algorithm—a gradient-based search method using continuous relaxation. This allows for the simultaneous evaluation of multiple candidate architectures through backpropagation, rather than training individual candidates from scratch.
Experimental Results
We evaluated LD-NAS across 8 classification tasks derived from MNIST (Tasks 0-3) and CIFAR-10 (Tasks 4-7). LD-NAS consistently outperformed or matched state-of-the-art methods like DARTS and ENAS in terms of accuracy, model size, and search speed.
On Task 6, LD-NAS achieved an accuracy of 90.87% with only 3.02 million parameters, making it more compact than DARTS (3.12M) and significantly leaner than VGG-16 (14.72M). Crucially, the search was completed in just 2 GPU days, matching the speed of DARTS while delivering a more efficient model. On MNIST (Task 3), LD-NAS reached 99.67% accuracy, surpassing both DARTS (99.51%) and ENAS (97.77%).
Conclusion
LD-NAS demonstrates that leveraging task similarity can significantly optimize the NAS process. By reusing architectural knowledge, we reduce search time to a fraction of traditional methods without sacrificing accuracy. Future work will explore the application of this framework to larger datasets like ImageNet and non-vision domains such as Natural Language Processing, as well as refining the FIM approximation to capture parameter correlations.
Summary updated