Is Support Set Diversity Necessary for Meta-Learning?

Amrith Setlur, Oscar Li, Virginia Smith

Abstract

Conventional wisdom in meta-learning posits that high task diversity during training is crucial for robust generalization. This paper challenges this assumption by introducing FIX-ML, a training regime that uses a fixed, static support pool for each class. Experimental results on standard benchmarks like miniImageNet and CIFAR-FS show that FIX-ML consistently improves performance, particularly for over-parameterized models. The authors find that while FIX-ML leads to higher training loss, it significantly reduces the generalization gap, suggesting that excessive task diversity may introduce noise that hinders optimization in deep neural networks.

Introduction

Meta-learning, or "learning to learn," typically employs episodic training where models are exposed to a vast array of tasks, each defined by a support set and a query set. The prevailing belief is that sampling from the astronomical number of possible support pools (e.g., $3 \times 10^{755}$ for miniImageNet) is necessary for the model to learn to generalize to unseen classes. This work investigates whether such extreme diversity is truly beneficial or if it introduces detrimental noise to the optimization process.

Methods

The authors propose FIX-ML. In standard meta-learning (ML), support and query sets are randomly sampled for every training episode. In FIX-ML, a small, static "support pool" is randomly selected for each class before training begins. Throughout training, whenever a class is included in a task, the model uses the same fixed images as the support set. Query sets remain randomly sampled to maintain the requirement for generalization. This approach was evaluated using several popular meta-learning frameworks, including Prototypical Networks (PN) and MetaOptNet.

Results

FIX-ML demonstrates superior performance across several benchmarks. On miniImageNet (5-way 5-shot) with a ResNet-12 backbone, FIX-ML achieved 78.19% accuracy, outperforming standard ML and several state-of-the-art methods like TADAM and MetaOptNet-SVM.

Figure 3: Performance comparison on miniImageNet and training loss curves. (a) FIX-ML outperforms standard ML and other baselines. (b, c) Training loss trajectories on CIFAR-FS showing the model's convergence on the fixed support set.

Analysis: Generalization and Model Capacity

The benefits of FIX-ML are most pronounced in over-parameterized architectures like ResNet-12 and WideResNet-16-10. For shallower models like Conv64, the advantage is less clear, suggesting that fixing the support set acts as a regularizer for high-capacity models that might otherwise overfit to the noise of changing tasks.

Interestingly, FIX-ML results in a higher training loss compared to standard ML when evaluated across all possible tasks, but it maintains a significantly smaller generalization gap. The authors found that the Takeuchi Information Criterion (TIC), a metric related to the curvature of the loss landscape, was a better predictor of this success than traditional flatness metrics.

Figure 6: Generalization analysis. (a, b) Loss landscape interpolations between FIX-ML and standard ML solutions. (d) Metrics showing FIX-ML reduces the generalization gap and improves the TIC across different datasets.

Conclusion

This research suggests that task diversity in meta-learning is not always beneficial. By restricting support set variety, FIX-ML provides a more stable optimization target for deep models, leading to better generalization on unseen tasks. This finding challenges core assumptions in the field and opens new avenues for regularizing meta-learning algorithms.

Summary updated

Abstract

Meta-learning is a popular framework for learning with limited data in which an algorithm is produced by training over multiple few-shot learning tasks. For classification problems, these tasks are typically constructed by sampling a small number of support and query examples from a subset of the classes. While conventional wisdom is that task diversity should improve the performance of meta-learning, in this work we find evidence to the contrary: we propose a modification to traditional meta-learning approaches in which we keep the support sets fixed across tasks, thus reducing task diversity. Surprisingly, we find that not only does this modification not result in adverse effects, it almost always improves the performance for a variety of datasets and meta-learning methods. We also provide several initial analyses to understand this phenomenon. Our work serves to: (i) more closely investigate the effect of support set construction for the problem of meta-learning, and (ii) suggest a simple, general, and competitive baseline for few-shot learning.
BibTeX
@misc{setlur2020is,
  author = {Amrith Setlur and Oscar Li and Virginia Smith},
  title = {{Is Support Set Diversity Necessary for Meta-Learning?}},
  year = {2020},
  eprint = {2011.14048},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.14048},
}

Loading related content...

Figure