Abstract
In this paper, we propose a neural architecture search (NAS) framework based on a similarity measure between baseline tasks and a target task. We define task similarity using the log-determinant of the Fisher Information Matrix (FIM). By utilizing the relationship between a target and a set of learned baseline tasks, the search space for the target task can be significantly reduced. This makes the discovery of optimal architectures tractable and efficient in terms of GPU days. Our method, LD-NAS, eliminates the requirement for training networks from scratch and reduces human domain bias in search space initialization.
Introduction
Neural Architecture Search (NAS) has revolutionized the design of deep learning models, often surpassing human-engineered architectures. However, widespread adoption is hindered by two primary challenges: extreme computational costs—often requiring thousands of GPU days—and a persistent reliance on human-defined search spaces, which can limit the discovery of truly novel designs.
We argue that the current paradigm of starting NAS from scratch for every new task is inefficient. Drawing from transfer learning principles, we propose that similar tasks should benefit from similar architectures. The core contribution of this work is a mathematical framework to define "task similarity" and use it to guide and prune the architectural search process.
Methodology
Task Similarity via Fisher Information
To quantify the distance between a baseline task ($b$) and a target task ($t$), we utilize the log-determinant of the Fisher Information Matrix (FIM). The FIM captures the sensitivity of a model's loss function to its parameters.
Crucially, our distance measure is asymmetric ($d[b, t] \neq d[t, b]$). This asymmetry reflects the hierarchical nature of knowledge transfer: it is generally more effective to transfer knowledge from a complex, high-entropy task to a simpler one than vice versa. To maintain computational efficiency, we approximate the FIM using its diagonal entries, reducing complexity from $O(n^2)$ to $O(n)$.
Reduced Search Space and Fusion Search
Once the most similar baseline task is identified, its architecture serves as the foundation for a restricted search space for the target task. We then employ the Fusion Search (FUSE) algorithm, a gradient-based method that uses continuous relaxation to evaluate multiple candidate architectures simultaneously, avoiding the need for exhaustive individual training.
Experimental Setup
We evaluated LD-NAS across eight classification tasks derived from standard benchmarks:
- MNIST (Tasks 0–3): Ranging from binary digit classification to full 10-digit classification.
- CIFAR-10 (Tasks 4–7): Including binary and multi-class subsets of natural images.
- VGG-16 served as the base approximation network for calculating the initial Fisher Information matrices.
Key Insights
- Complexity over Content: Interestingly, Task 7 (CIFAR-10) was found to be the closest baseline for Task 3 (MNIST 10-class). This suggests that structural task complexity (e.g., number of classes) can be a more significant factor for architectural suitability than the specific visual domain of the data.
- The Power of Asymmetry: By acknowledging that transfer is a "one-way street," LD-NAS avoids using overly simplistic models as templates for complex tasks, a common pitfall in symmetric distance measures.
- Efficiency through Pruning: Starting with a "warm" search space based on a related task prevents the algorithm from wasting cycles on architectures that are fundamentally mismatched to the target task's complexity.
Results
LD-NAS demonstrates superior performance and efficiency compared to state-of-the-art NAS methods like ENAS and DARTS.
As shown in the results for Task 6, LD-NAS achieved an accuracy of 90.87% while utilizing only 3.02 million parameters. This is significantly leaner than standard architectures like VGG-16 (14.72M) and more accurate than ENAS (75.22%). Furthermore, the search process required only 2 GPU days, matching the speed of DARTS while delivering higher accuracy and smaller model sizes.
Conclusion and Future Work
LD-NAS successfully leverages transfer learning to make automated machine learning more accessible and efficient. While the current framework relies on a library of baseline tasks and diagonal FIM approximations, it sets a new standard for parameter-efficient NAS. Future research will explore scaling this approach to larger datasets like ImageNet and expanding into the domain of Natural Language Processing (NLP).
Summary updated