Fisher Task Distance and Its Applications in Transfer Learning and Neural Architecture Search.

Cat P. Le, Mohammadreza Soltani, Juncheng Dong, Vahid TarokharXiv 2021 · arXiv

Abstract

We formulate an asymmetric (or non-commutative) distance between tasks based on Fisher Information Matrices. We provide proof of consistency for our distance through theorems and experiments on various classification tasks. We then apply our proposed measure of task distance in transfer learning on visual tasks in the Taskonomy dataset. Additionally, we show how the proposed distance between a target task and a set of baseline tasks can be used to reduce the neural architecture search space for the target task. The complexity reduction in search space for task-specific architectures is achieved by building on the optimized architectures for similar tasks instead of doing a full search without using this side information. Experimental results demonstrate the efficacy of the proposed approach and its improvements over other methods.

Introduction

In the field of deep learning, the intuition that "similar tasks require similar architectures" is widely held but difficult to quantify. Traditionally, researchers have relied on domain expertise to guess task relationships or employed brute-force transfer learning—training models on every possible pair of tasks to empirically determine compatibility. However, these methods are either subjective or computationally prohibitive.

Furthermore, Neural Architecture Search (NAS), which automates the design of neural networks, often requires thousands of GPU hours because it typically starts from scratch for every new task. This research addresses these gaps by introducing the Fisher Task Distance (FTD), a mathematically grounded, asymmetric metric for task similarity, and leveraging it to create a "Task-Aware" Neural Architecture Search (TA-NAS) framework that significantly reduces computational costs.

The Fisher Task Distance (FTD)

The core contribution of this work is the Fisher Task Distance. Unlike standard distance metrics, FTD is asymmetric ($d(a, b) \neq d(b, a)$). This asymmetry is crucial because it reflects the reality of transfer learning: it is often easier to transfer knowledge from a complex, high-entropy task (like ImageNet classification) to a simpler one (like binary classification) than the reverse.

Technical Formulation

FTD is calculated using the Fisher Information Matrix (FIM), which represents the second derivative of the loss function with respect to model parameters. To ensure computational feasibility, the authors utilize a diagonal approximation of the FIM. Theoretical proofs (Theorems 1 and 2 in the paper) establish that FTD is a well-defined measure that converges consistently regardless of initial settings or learning rates.

One of the most robust findings is that the relative distance between tasks remains consistent across different "probe" networks (e.g., VGG-16 vs. ResNet-18). This architecture independence suggests that FTD captures fundamental semantic relationships between tasks rather than artifacts of a specific model structure.

Task-Aware Neural Architecture Search (TA-NAS)

The TA-NAS framework utilizes FTD to "warm-start" the search process. Instead of searching the entire universe of possible architectures, the system identifies the most similar task from a library of previously solved "baseline" tasks and restricts the search to a neighborhood around that task's optimized architecture.

A representative cell structure within the neural architecture search space, illustrating various operations between nodes.

The pipeline consists of three primary steps:

  1. Distance Calculation: Compute the FTD between the new target task and a library of baseline tasks.
  2. Search Space Reduction: Identify the closest baseline task and restrict the architecture search space to configurations similar to that task's optimal architecture.
  3. FUSE Algorithm: Employ a gradient-based "Fusion Search" to refine and find the final optimal architecture within the restricted space.

Experimental Evaluation

The authors validated FTD and TA-NAS across several benchmarks, including MNIST, CIFAR-10/100, ImageNet, and the Taskonomy dataset.

Key Results

  • ImageNet Efficiency: TA-NAS identified an architecture achieving 95.92% accuracy with only 3.43 million parameters in 4 GPU days. This outperformed standard DARTS, which achieved 95.22% accuracy in a similar timeframe.
  • CIFAR-100 Sub-tasks: On specific 11-class sub-tasks, TA-NAS reached 90.96% accuracy, significantly surpassing ENAS (10.49%) and DARTS (87.57%).
  • Taskonomy Alignment: The FTD metric produced a task map that closely mirrored results from expensive brute-force methods. It correctly identified semantic relationships, such as the proximity between 2D-edge detection and 3D-keypoint detection.

Conclusion and Future Work

Fisher Task Distance provides a scalable and theoretically sound method for mapping the relationship between machine learning tasks. By using this distance to guide Neural Architecture Search, the TA-NAS framework demonstrates that we can achieve state-of-the-art performance with a fraction of the typical search complexity.

While the current method relies on a diagonal FIM approximation and assumes a degree of convexity for its proofs, future work may explore block-diagonal approximations to capture parameter correlations. Additionally, expanding the library of baseline tasks will further enhance the framework's ability to generalize to increasingly diverse target tasks.

Summary updated

Abstract

We formulate an asymmetric (or non-commutative) distance between tasks based on Fisher Information Matrices. We provide proof of consistency for our distance through theorems and experiments on various classification tasks. We then apply our proposed measure of task distance in transfer learning on visual tasks in the Taskonomy dataset. Additionally, we show how the proposed distance between a target task and a set of baseline tasks can be used to reduce the neural architecture search space for the target task. The complexity reduction in search space for task-specific architectures is achieved by building on the optimized architectures for similar tasks instead of doing a full search without using this side information. Experimental results demonstrate the efficacy of the proposed approach and its improvements over other methods.
BibTeX
@misc{le2021fisher,
  author = {Cat P. Le and Mohammadreza Soltani and Juncheng Dong and Vahid Tarokh},
  title = {{Fisher Task Distance and Its Applications in Transfer Learning and Neural Architecture Search.}},
  year = {2021},
  eprint = {2103.12827},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.12827},
}
Figure