Improved Automated Machine Learning from Transfer Learning

Cat P. Le, Mohammadreza Soltani, Robert Ravier, Vahid Tarokh

Abstract

In this paper, we propose a neural architecture search (NAS) framework based on a similarity measure between baseline tasks and a target task. We define task similarity using the log-determinant of the Fisher Information Matrix (FIM). By utilizing the relationship between a target and a set of learned baseline tasks, the search space for the target task can be significantly reduced. This makes the discovery of optimal architectures tractable and efficient in terms of GPU days. Our method, LD-NAS, eliminates the requirement for training networks from scratch and reduces human domain bias in search space initialization.

Introduction

Neural Architecture Search (NAS) has revolutionized the design of deep learning models, often surpassing human-engineered architectures. However, widespread adoption is hindered by two primary challenges: extreme computational costs—often requiring thousands of GPU days—and a persistent reliance on human-defined search spaces, which can limit the discovery of truly novel designs.

We argue that the current paradigm of starting NAS from scratch for every new task is inefficient. Drawing from transfer learning principles, we propose that similar tasks should benefit from similar architectures. The core contribution of this work is a mathematical framework to define "task similarity" and use it to guide and prune the architectural search process.

Methodology

Task Similarity via Fisher Information

To quantify the distance between a baseline task ($b$) and a target task ($t$), we utilize the log-determinant of the Fisher Information Matrix (FIM). The FIM captures the sensitivity of a model's loss function to its parameters.

Crucially, our distance measure is asymmetric ($d[b, t] \neq d[t, b]$). This asymmetry reflects the hierarchical nature of knowledge transfer: it is generally more effective to transfer knowledge from a complex, high-entropy task to a simpler one than vice versa. To maintain computational efficiency, we approximate the FIM using its diagonal entries, reducing complexity from $O(n^2)$ to $O(n)$.

Task similarity matrices showing mean (a) and standard deviation (b) of distances between source and target tasks.

Once the most similar baseline task is identified, its architecture serves as the foundation for a restricted search space for the target task. We then employ the Fusion Search (FUSE) algorithm, a gradient-based method that uses continuous relaxation to evaluate multiple candidate architectures simultaneously, avoiding the need for exhaustive individual training.

Experimental Setup

We evaluated LD-NAS across eight classification tasks derived from standard benchmarks:

  • MNIST (Tasks 0–3): Ranging from binary digit classification to full 10-digit classification.
  • CIFAR-10 (Tasks 4–7): Including binary and multi-class subsets of natural images.
  • VGG-16 served as the base approximation network for calculating the initial Fisher Information matrices.

Key Insights

  • Complexity over Content: Interestingly, Task 7 (CIFAR-10) was found to be the closest baseline for Task 3 (MNIST 10-class). This suggests that structural task complexity (e.g., number of classes) can be a more significant factor for architectural suitability than the specific visual domain of the data.
  • The Power of Asymmetry: By acknowledging that transfer is a "one-way street," LD-NAS avoids using overly simplistic models as templates for complex tasks, a common pitfall in symmetric distance measures.
  • Efficiency through Pruning: Starting with a "warm" search space based on a related task prevents the algorithm from wasting cycles on architectures that are fundamentally mismatched to the target task's complexity.

Results

LD-NAS demonstrates superior performance and efficiency compared to state-of-the-art NAS methods like ENAS and DARTS.

Performance comparison of LD-NAS against baseline architectures and other NAS methods on CIFAR-10 Task 6.

As shown in the results for Task 6, LD-NAS achieved an accuracy of 90.87% while utilizing only 3.02 million parameters. This is significantly leaner than standard architectures like VGG-16 (14.72M) and more accurate than ENAS (75.22%). Furthermore, the search process required only 2 GPU days, matching the speed of DARTS while delivering higher accuracy and smaller model sizes.

Conclusion and Future Work

LD-NAS successfully leverages transfer learning to make automated machine learning more accessible and efficient. While the current framework relies on a library of baseline tasks and diagonal FIM approximations, it sets a new standard for parameter-efficient NAS. Future research will explore scaling this approach to larger datasets like ImageNet and expanding into the domain of Natural Language Processing (NLP).

Summary updated

Abstract

In this paper, we propose a neural architecture search framework based on a similarity measure between some baseline tasks and a target task. We first define the notion of the task similarity based on the log-determinant of the Fisher Information matrix. Next, we compute the task similarity from each of the baseline tasks to the target task. By utilizing the relation between a target and a set of learned baseline tasks, the search space of architectures for the target task can be significantly reduced, making the discovery of the best candidates in the set of possible architectures tractable and efficient, in terms of GPU days. This method eliminates the requirement for training the networks from scratch for a given target task as well as introducing the bias in the initialization of the search space from the human domain.
BibTeX
@misc{le2021improved,
  author = {Cat P. Le and Mohammadreza Soltani and Robert Ravier and Vahid Tarokh},
  title = {{Improved Automated Machine Learning from Transfer Learning}},
  year = {2021},
  eprint = {2103.00241},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.00241},
}
Figure