Task similarity aware meta learning: theory-inspired improvement on MAML

Pan Zhou, Yingtian Zou, Xiao-Tong Yuan, Jiashi Feng, Caiming Xiong, Steven C. H. HoiUAI 2021 · OpenReview

Abstract

Few-shot learning ability is heavily desired for machine intelligence. By meta-learning a model initialization from training tasks with fast adaptation ability to new tasks, model-agnostic meta-learning (MAML) has achieved great success. However, theoretical understanding of its generalization and its limitation in handling diverse task distributions remained open. This paper provides a rigorous theoretical justification for MAML's success and introduces TSA-MAML, a variant that learns multiple, group-specific initializations to better handle diverse tasks.

Introduction

Few-shot learning aims to train models that can learn new concepts from just a handful of examples. Model-Agnostic Meta-Learning (MAML) has become the industry standard for this, finding a "meta-initialization"—a starting point for model weights that can adapt to a new task with just 1–5 steps of gradient descent.

Despite its success, two major gaps persisted:

  1. Theoretical Justification: There was no solid proof explaining why a few gradient steps on tiny datasets avoid massive overfitting.
  2. The "One-Size-Fits-All" Limitation: MAML learns a single initialization for all tasks. For heterogeneous task sets (e.g., birds vs. aircraft), a single starting point is often a mediocre compromise, slowing down adaptation.

Theoretical Foundation

The authors provide the first rigorous theoretical justification for MAML. They prove that the excess risk (the performance gap between the adapted model and the theoretical optimum) is bounded by the Euclidean distance between the meta-initialization ($\theta^*$) and the task’s true optimal parameters ($\theta^*_T$). This confirms that MAML’s success is mathematically tied to its ability to find a central starting point in the parameter space.

TSA-MAML: Task Similarity Aware Meta-Learning

Inspired by this theory, the authors propose TSA-MAML. Instead of one initialization, TSA-MAML learns $m$ different initializations to better cover the parameter space of diverse tasks.

Parameter Space Clustering

A key insight of the paper is that task similarity is better captured in the parameter space than the feature space. As shown in Figure 1, while image features of different datasets (Birds, Fungi, Aircraft) often overlap (a), the optimal model parameters for those same datasets form distinct, separable clusters (b). TSA-MAML leverages this by clustering tasks based on their adapted weights rather than raw features.

Figure 1: t-SNE visualization of feature space vs. parameter space clustering. TSA-MAML (c) uses multiple initializations to better match task clusters compared to vanilla MAML (b).

The Algorithm

  1. Probing: A "learner" (vanilla MAML) estimates the optimal parameters for a given task.
  2. Clustering: Tasks are clustered in the parameter space.
  3. Group-Specific Initialization: During training, a task is assigned to the closest initialization group. During testing, the model "probes" the new task to select the best initialization group before final fine-tuning.

Experimental Results

The authors evaluated TSA-MAML on group-structured datasets (e.g., Aircraft + Birds + Fungi) and standard benchmarks like CIFARFS and miniImageNet.

Performance Gains

On the heterogeneous Aircraft/Birds/Fungi dataset, TSA-MAML improved accuracy by approximately 4% over vanilla MAML. On CIFARFS (5-shot 5-way), it achieved 71.37% accuracy, significantly outperforming MAML's 68.91%.

Figure 6 illustrates the optimization process, showing that TSA-MAML achieves a lower test loss more quickly and maintains better stability than standard MAML.

Figure 6: Test loss comparison between MAML and TSA-MAML over iterations in a 1-shot 10-way setting.

Discussion and Insights

  • Distance is Destiny: Adaptation speed and accuracy are functions of Euclidean distance in weight space. Shortening this distance by using multiple initializations directly improves performance.
  • The Overfitting Paradox: While more gradient steps usually help reach the optimum, they also increase noise from small training sets. TSA-MAML mitigates this by starting closer to the optimum, requiring fewer steps.
  • Limitations: TSA-MAML introduces computational overhead due to the "probing" step and requires pre-defining the number of clusters ($m$). Future work could focus on automating cluster discovery and scaling to larger foundation models.

Summary updated

Abstract

Few-shot learning ability is heavily desired for machine intelligence. By meta-learning a model initialization from training tasks with fast adaptation ability to new tasks, model-agnostic meta-lea...
BibTeX
@inproceedings{zhou2021task,
  author = {Pan Zhou and Yingtian Zou and Xiao-Tong Yuan and Jiashi Feng and Caiming Xiong and Steven C. H. Hoi},
  title = {{Task similarity aware meta learning: theory-inspired improvement on MAML}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=aqNIyAhbgT},
}
Figure