Neural Architecture Search From Task Similarity Measure

Cat P. Le, Mohammadreza Soltani, Robert J. Ravier, Vahid Tarokh

Abstract

In this paper, we propose a neural architecture search (NAS) framework based on a similarity measure between baseline tasks and a target task. We define task similarity using the log-determinant of the Fisher Information Matrix (FIM). By quantifying the relationship between a target and a set of learned baseline tasks, the search space for the target task can be significantly reduced. This approach makes the discovery of optimal architectures tractable and efficient, requiring only 2 GPU days. Our method eliminates the need for training networks from scratch for new tasks and reduces human bias in search space initialization.

Introduction

Neural Architecture Search (NAS) represents a significant step toward fully automated machine learning, yet it remains computationally prohibitive, often requiring thousands of GPU days. Traditional NAS methods typically suffer from two limitations: they ignore knowledge from previous similar tasks (starting from zero) and rely on human-defined search spaces that may limit architectural diversity.

We introduce LD-NAS, a framework that bridges transfer learning and NAS. By identifying which previously solved tasks are most similar to a new target task, LD-NAS reuses proven architectural search spaces, drastically reducing the computational overhead while maintaining or improving performance.

Methodology

Task Similarity via Fisher Information

The core of LD-NAS is a dissimilarity measure $d[b, t]$ based on the log-determinant of the Fisher Information Matrix (FIM). The FIM captures how much information the data provides about model parameters. To ensure computational tractability, we utilize a diagonal approximation of the FIM, reducing the complexity from $n^2$ to $n$ entries.

Mean task similarity distances between source and target tasks. Lower values indicate higher similarity.

As shown in the similarity heatmap, this distance is asymmetric; transferring knowledge from a complex task to a simpler one is often more effective than the reverse. Interestingly, structural complexity often outweighs domain similarity. For instance, Task 7 (CIFAR-10) was found to be a close match for Task 3 (MNIST), as both involve 10-class classification, despite the visual differences between natural objects and grayscale digits.

Search Space Reduction and Optimization

Once the most similar "base task" is identified, LD-NAS adopts its architectural search space. We then apply the FUSE algorithm—a gradient-based search method using continuous relaxation. This allows for the simultaneous evaluation of multiple candidate architectures through backpropagation, rather than training individual candidates from scratch.

Experimental Results

We evaluated LD-NAS across 8 classification tasks derived from MNIST (Tasks 0-3) and CIFAR-10 (Tasks 4-7). LD-NAS consistently outperformed or matched state-of-the-art methods like DARTS and ENAS in terms of accuracy, model size, and search speed.

Performance comparison of LD-NAS against baseline architectures and search methods on Task 6 (CIFAR-10 subset).

On Task 6, LD-NAS achieved an accuracy of 90.87% with only 3.02 million parameters, making it more compact than DARTS (3.12M) and significantly leaner than VGG-16 (14.72M). Crucially, the search was completed in just 2 GPU days, matching the speed of DARTS while delivering a more efficient model. On MNIST (Task 3), LD-NAS reached 99.67% accuracy, surpassing both DARTS (99.51%) and ENAS (97.77%).

Conclusion

LD-NAS demonstrates that leveraging task similarity can significantly optimize the NAS process. By reusing architectural knowledge, we reduce search time to a fraction of traditional methods without sacrificing accuracy. Future work will explore the application of this framework to larger datasets like ImageNet and non-vision domains such as Natural Language Processing, as well as refining the FIM approximation to capture parameter correlations.

Summary updated

Abstract

In this paper, we propose a neural architecture search framework based on a similarity measure between some baseline tasks and a target task. We first define the notion of the task similarity based on the log-determinant of the Fisher Information matrix. Next, we compute the task similarity from each of the baseline tasks to the target task. By utilizing the relation between a target and a set of learned baseline tasks, the search space of architectures for the target task can be significantly reduced, making the discovery of the best candidates in the set of possible architectures tractable and efficient, in terms of GPU days. This method eliminates the requirement for training the networks from scratch for a given target task as well as introducing the bias in the initialization of the search space from the human domain.
BibTeX
@misc{le2021neural,
  author = {Cat P. Le and Mohammadreza Soltani and Robert J. Ravier and Vahid Tarokh},
  title = {{Neural Architecture Search From Task Similarity Measure}},
  year = {2021},
  eprint = {2103.00241},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.00241},
}
Figure