Abstract
Conventional wisdom in meta-learning posits that high task diversity during training is crucial for robust generalization. This paper challenges this assumption by introducing FIX-ML, a training regime that uses a fixed, static support pool for each class. Experimental results on standard benchmarks like miniImageNet and CIFAR-FS show that FIX-ML consistently improves performance, particularly for over-parameterized models. The authors find that while FIX-ML leads to higher training loss, it significantly reduces the generalization gap, suggesting that excessive task diversity may introduce noise that hinders optimization in deep neural networks.
Introduction
Meta-learning, or "learning to learn," typically employs episodic training where models are exposed to a vast array of tasks, each defined by a support set and a query set. The prevailing belief is that sampling from the astronomical number of possible support pools (e.g., $3 \times 10^{755}$ for miniImageNet) is necessary for the model to learn to generalize to unseen classes. This work investigates whether such extreme diversity is truly beneficial or if it introduces detrimental noise to the optimization process.
Methods
The authors propose FIX-ML. In standard meta-learning (ML), support and query sets are randomly sampled for every training episode. In FIX-ML, a small, static "support pool" is randomly selected for each class before training begins. Throughout training, whenever a class is included in a task, the model uses the same fixed images as the support set. Query sets remain randomly sampled to maintain the requirement for generalization. This approach was evaluated using several popular meta-learning frameworks, including Prototypical Networks (PN) and MetaOptNet.
Results
FIX-ML demonstrates superior performance across several benchmarks. On miniImageNet (5-way 5-shot) with a ResNet-12 backbone, FIX-ML achieved 78.19% accuracy, outperforming standard ML and several state-of-the-art methods like TADAM and MetaOptNet-SVM.
Analysis: Generalization and Model Capacity
The benefits of FIX-ML are most pronounced in over-parameterized architectures like ResNet-12 and WideResNet-16-10. For shallower models like Conv64, the advantage is less clear, suggesting that fixing the support set acts as a regularizer for high-capacity models that might otherwise overfit to the noise of changing tasks.
Interestingly, FIX-ML results in a higher training loss compared to standard ML when evaluated across all possible tasks, but it maintains a significantly smaller generalization gap. The authors found that the Takeuchi Information Criterion (TIC), a metric related to the curvature of the loss landscape, was a better predictor of this success than traditional flatness metrics.
Conclusion
This research suggests that task diversity in meta-learning is not always beneficial. By restricting support set variety, FIX-ML provides a more stable optimization target for deep models, leading to better generalization on unseen tasks. This finding challenges core assumptions in the field and opens new avenues for regularizing meta-learning algorithms.
Summary updated