Abstract
We categorize meta-learning evaluation into two settings: $\textit{in-distribution}$ [ID], in which the train and test tasks are sampled $\textit{iid}$ from the same underlying task distribution, and $\textit{out-of-distribution}$ [OOD], in which they are not. While most meta-learning theory and some FSL applications follow the ID setting, we identify that most existing few-shot classification benchmarks instead reflect OOD evaluation, as they use disjoint sets of train (base) and test (novel) classes for task generation. This discrepancy is problematic because -- as we show on numerous benchmarks -- meta-learning methods that perform better on existing OOD datasets may perform significantly worse in the ID setting. In addition, in the OOD setting, even though current FSL benchmarks seem befitting, our study highlights concerns in 1) reliably performing model selection for a given meta-learning method, and 2) consistently comparing the performance of different methods. To address these concerns, we provide suggestions on how to construct FSL benchmarks to allow for ID evaluation as well as more reliable OOD evaluation. Our work aims to inform the meta-learning community about the importance and distinction of ID vs. OOD evaluation, as well as the subtleties of OOD evaluation with current benchmarks.
Introduction
Meta-learning, or "learning to learn," focuses on developing algorithms capable of solving new tasks with limited examples. A fundamental assumption in meta-learning theory is that training and test tasks are sampled from the same underlying distribution—an "in-distribution" (ID) setting. However, in practice, most few-shot learning (FSL) benchmarks, such as miniImageNet, utilize disjoint sets of classes for training and testing, effectively evaluating "out-of-distribution" (OOD) generalization. This paper investigates the implications of this discrepancy, revealing that optimizing for OOD performance can come at the cost of ID performance and that current OOD evaluation protocols are statistically fragile.
Methodology
To systematically study the ID vs. OOD trade-off, the authors introduced "Mod" versions of standard benchmarks (e.g., mini-M, tiered-M, cifar-M). These datasets allow for the simultaneous measurement of BaseGen (ID performance on held-out images of training classes) and NovelGen (OOD performance on entirely new classes). The study evaluated four prominent meta-learning methods: Prototypical Networks (PN), MetaOptNet (using SVM and Ridge heads), and First-Order MAML (FOMAML). Additionally, the authors analyzed the reliability of model selection via validation sets and the statistical significance of benchmark rankings through "conclusion flip" simulations.
The ID vs. OOD Performance Gap
A central finding of this work is the "performance flip": algorithms that excel in ID settings often underperform in OOD settings, and vice versa. For example, on the Zappos-ID benchmark, Prototypical Networks (PN) outperform SVM-based methods by over 3%. However, on OOD benchmarks like miniImageNet, the ranking reverses, with SVM outperforming PN by 1.5%.
As shown in Figure 1, the relative rankings of methods are highly sensitive to the evaluation setting. This suggests that the meta-learning community may be inadvertently favoring algorithms that generalize well to new classes at the expense of their ability to master the distribution they were trained on. Furthermore, common "tricks" like training with more "ways" than the test task often boost OOD scores while actively harming ID performance.
Reliability of Model Selection and Evaluation
Beyond performance trade-offs, the paper highlights systemic issues in how meta-learning models are selected and compared. In standard supervised learning, validation performance is a reliable proxy for test performance. In OOD meta-learning, however, the validation classes differ from the test classes, leading to a breakdown in this relationship.
The correlation analysis in Figure 2 reveals that validation performance often fails to track test performance, sometimes even exhibiting negative correlation. This leads to a "model selection loss," where choosing a snapshot based on OOD validation results in a performance drop of up to 1.2% compared to the true best snapshot.
Furthermore, the small number of novel classes in standard benchmarks (typically 20) introduces high statistical variance. The authors' "conclusion flip" analysis shows a ~15% probability that a reported performance advantage of one algorithm over another is merely an artifact of the specific test set used, rather than a true reflection of algorithmic superiority.
Conclusion and Recommendations
This research serves as a cautionary tale for the meta-learning community. The current focus on OOD benchmarks without considering ID performance provides an incomplete picture of algorithmic progress. To improve the field's rigor, the authors recommend reporting both BaseGen (ID) and NovelGen (OOD) scores, increasing the number of novel classes in benchmarks to reduce statistical noise, and developing more robust model selection strategies. By rethinking how we evaluate meta-learning, we can develop more robust algorithms that balance specialized mastery with broad generalization.
Summary updated