TADAM: Task dependent adaptive metric for improved few-shot learning

Boris N. Oreshkin, Pau Rodriguez, Alexandre Lacoste

Abstract

Few-shot learning has become essential for producing models that generalize from few examples. In this work, we identify that metric scaling and metric task conditioning are important to improve the performance of few-shot algorithms. Our analysis reveals that simple metric scaling completely changes the nature of few-shot algorithm parameter updates. Metric scaling provides improvements up to 14% in accuracy for certain metrics on the mini-Imagenet 5-way 5-shot classification task. We further propose a simple and effective way of conditioning a learner on the task sample set, resulting in learning a task-dependent metric space. Moreover, we propose and empirically test a practical end-to-end optimization procedure based on auxiliary task co-training to learn a task-dependent metric space. The resulting few-shot learning model based on the task-dependent scaled metric achieves state of the art on mini-Imagenet. We confirm these results on another few-shot dataset that we introduce in this paper based on CIFAR100.

Introduction

Few-shot learning (FSL) aims to train models capable of recognizing new categories from a very limited number of examples. While metric learning approaches—where models learn to map images into a space where similar items are clustered—have been successful, they often rely on a static feature extractor. This means the model looks for the same features regardless of the specific task, such as distinguishing between dog breeds versus types of fruit.

TADAM (Task-Dependent Adaptive Metric) addresses this limitation by making the feature extractor "aware" of the specific task. By dynamically adapting the model's "lens" based on the support set, TADAM significantly boosts performance across standard benchmarks.

Methods

The TADAM framework introduces three primary innovations to the Prototypical Network architecture:

  1. Metric Scaling ($\alpha$): The authors found that scaling the distance metric (Euclidean or Cosine) by a learnable temperature parameter $\alpha$ before the softmax layer is critical. Mathematical analysis shows that $\alpha$ controls the optimization focus: small values reduce overall distribution overlap, while large values prioritize correcting errors on "hard" examples.
  2. Task Conditioning (TEN): A Task Encoding Network (TEN) processes the support set to calculate a task representation (the mean of class prototypes). This representation generates parameters for FiLM (Feature-wise Linear Modulation) layers, which apply task-specific scale ($\gamma$) and shift ($\beta$) transformations to the feature extractor’s internal activations.
  3. Auxiliary Task Co-training: To stabilize the complex optimization of a task-dependent model, the system is co-trained on a standard classification task (e.g., 64-way classification for mini-ImageNet) alongside few-shot episodes. This ensures the feature extractor retains general visual knowledge.
The TADAM architecture featuring the Task Encoding Network (TEN) and FiLM-based conditioning.

Datasets

The model was evaluated on two key datasets:

  • mini-ImageNet: The standard FSL benchmark derived from ImageNet.
  • FC100 (Few-shot CIFAR100): A more challenging dataset introduced in this paper. It is split by superclasses (e.g., "fish" vs. "flowers"), preventing the model from relying on low-level features shared between similar classes during training.

Key Insights and Ideas

The Scaling Effect

A significant finding is that the perceived superiority of Euclidean distance over Cosine similarity in previous FSL research was largely an artifact of scaling. When Cosine similarity is properly scaled by $\alpha$, the performance gap disappears. Furthermore, the scale of the metric fundamentally alters parameter updates, suggesting that "temperature" is a structural component of the optimization logic rather than just a tuning parameter.

Impact of the metric scale parameter α on validation and test accuracy.

Layer-wise Task Specificity

Ablation studies revealed that task conditioning is most effective in the deeper layers of the network. This aligns with the intuition that low-level features like edges are universal, whereas high-level features representing complex shapes or parts benefit from being task-specific.

Key Results

TADAM achieved state-of-the-art results at the time of publication:

  • mini-ImageNet: Reached 58.5% (1-shot) and 76.7% (5-shot) accuracy, a substantial improvement over previous models like adaResNet (71.9% for 5-shot).
  • Ablation Impact: Metric scaling alone improved baseline Prototypical Networks by up to 14% in certain configurations.
  • FC100: Demonstrated consistent gains on this harder dataset, reaching 56.1% on 5-shot tasks.

Conclusion and Future Work

While TADAM sets a new bar for FSL, it introduces training complexities that require auxiliary co-training for stability. Future work could explore richer task representations beyond the mean of prototypes, such as incorporating variance or covariance. Additionally, the discovery of how $\alpha$ affects gradients suggests new directions for designing loss functions that adaptively focus on example difficulty during training.

Summary updated

Abstract

Few-shot learning has become essential for producing models that generalize from few examples. In this work, we identify that metric scaling and metric task conditioning are important to improve the performance of few-shot algorithms. Our analysis reveals that simple metric scaling completely changes the nature of few-shot algorithm parameter updates. Metric scaling provides improvements up to 14% in accuracy for certain metrics on the mini-Imagenet 5-way 5-shot classification task. We further propose a simple and effective way of conditioning a learner on the task sample set, resulting in learning a task-dependent metric space. Moreover, we propose and empirically test a practical end-to-end optimization procedure based on auxiliary task co-training to learn a task-dependent metric space. The resulting few-shot learning model based on the task-dependent scaled metric achieves state of the art on mini-Imagenet. We confirm these results on another few-shot dataset that we introduce in this paper based on CIFAR100. Our code is publicly available at this https URL.
BibTeX
@inproceedings{oreshkin2018tadam,
  author = {Boris N. Oreshkin and Pau Rodriguez and Alexandre Lacoste},
  title = {{TADAM: Task dependent adaptive metric for improved few-shot learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=rk-6xYb_Zr},
}
Figure