Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution

Amrith Setlur, Oscar Li, Virginia Smith

TL;DR

Most meta-learning research assumes that training and testing tasks come from the same distribution, yet the benchmarks used to evaluate them are almost exclusively "Out-of-Distribution" (OOD). This paper reveals a startling "ranking flip": methods that dominate standard benchmarks often perform significantly worse in "In-Distribution" (ID) settings. Furthermore, current OOD evaluation practices are prone to unreliable model selection and inconsistent results.

The Core Problem: A Distributional Mismatch

Meta-learning, or "learning to learn," aims to create algorithms that can solve new tasks quickly with very few examples. Traditionally, researchers evaluate these algorithms using Few-Shot Learning (FSL) benchmarks like miniImageNet.

The authors identify a fundamental mismatch in how these algorithms are evaluated. Meta-learning theory and many real-world applications (like federated learning or personalized recommendations) assume that training and testing tasks are sampled from the same underlying distribution—an In-Distribution (ID) setting. However, standard benchmarks use disjoint sets of classes for training ("base" classes) and testing ("novel" classes). This makes them inherently Out-of-Distribution (OOD) evaluations.

This discrepancy leads to three major issues:

  1. The ID/OOD Gap: We lack visibility into how well "state-of-the-art" meta-learners perform when the task distribution actually remains the same.
  2. Unreliable Model Selection: In OOD settings, using a validation set to pick the best model "snapshot" is highly unreliable because the validation tasks are also OOD.
  3. Inconsistent Comparisons: Because benchmarks use a small, fixed number of novel classes, the "winner" of a comparison can change simply by picking a different random subset of classes.

The "Ranking Flip"

To study the ID setting, the authors introduced realistic benchmarks (like FEMNIST and Zappos-ID) and created "Modified" versions of standard benchmarks (e.g., miniImageNet-Mod) that reserve a portion of the "base" class images to evaluate ID generalization.

When comparing performance across these settings, a clear "ranking flip" emerges. On standard OOD benchmarks like miniImageNet, methods like MetaOptNet-SVM often beat Prototypical Networks (PN). However, on every ID benchmark tested, PN significantly outperforms SVM.

BaseGen (ID) vs NovelGen (OOD) performance across different methods. Notice the crossing lines, indicating that methods excelling in OOD often perform worse in ID settings.

This suggests that "improving" OOD performance often comes at the direct expense of ID performance. Furthermore, common OOD training "tricks"—like training with more classes than used at test time—can boost OOD scores but actively degrade the model's ability to generalize to the training distribution.

Validation is a Poor Proxy

In standard machine learning, we use a validation set to choose the best training epoch. In OOD meta-learning, the authors used Kendall rank correlation ($\rho$) to quantify how well validation performance tracks with actual test performance across different training snapshots.

The correlation between validation accuracy and test accuracy in OOD settings is surprisingly low, and sometimes even negative.

Kendall rank correlation ($\rho$) between validation and test accuracy. The correlation in OOD meta-learning is vastly lower than in standard supervised learning (red dashed line).

Because validation is such a poor proxy, the common practice of "picking the best model on the val set" often selects a sub-optimal model for the final test. The authors found this standard practice resulted in a performance loss of up to 1.2% compared to the true best test snapshot—a massive margin in a field where papers are routinely published over 0.5% gains.

The "Conclusion Flip" and Statistical Instability

When an OOD benchmark only uses a small pool of novel classes (e.g., 20 classes), the evaluation is statistically unstable. The authors demonstrated that for a true performance difference of 0.5%, a standard 20-class evaluation results in "improvement exaggerations" 25% of the time. Furthermore, there is a high probability (up to 15%) that the performance order of two methods will flip entirely if evaluated on a different set of 20 classes. This calls into question the validity of many papers claiming marginal state-of-the-art improvements.

Moving Forward

The paper highlights a critical blind spot in meta-learning evaluation. To build more robust and reliable meta-learning systems, the authors suggest that future researchers should:

  1. Be Explicit: Clearly state whether a method targets ID or OOD generalization.
  2. Report Both: Report "BaseGen" (ID) performance alongside "NovelGen" (OOD) to transparently show the trade-offs.
  3. Scale Up Benchmarks: Use benchmarks with a much larger number of novel classes (like tieredImageNet or Meta-Dataset) to avoid statistical instability and "conclusion flips."
  4. Rethink Model Selection: Explore better model selection strategies that do not rely solely on disjoint OOD validation sets.

Summary updated

Abstract

We categorize meta-learning evaluation into two settings: $\textit{in-distribution}$ [ID], in which the train and test tasks are sampled $\textit{iid}$ from the same underlying task distribution, and $\textit{out-of-distribution}$ [OOD], in which they are not. While most meta-learning theory and some FSL applications follow the ID setting, we identify that most existing few-shot classification benchmarks instead reflect OOD evaluation, as they use disjoint sets of train (base) and test (novel) classes for task generation. This discrepancy is problematic because -- as we show on numerous benchmarks -- meta-learning methods that perform better on existing OOD datasets may perform significantly worse in the ID setting. In addition, in the OOD setting, even though current FSL benchmarks seem befitting, our study highlights concerns in 1) reliably performing model selection for a given meta-learning method, and 2) consistently comparing the performance of different methods. To address these concerns, we provide suggestions on how to construct FSL benchmarks to allow for ID evaluation as well as more reliable OOD evaluation. Our work aims to inform the meta-learning community about the importance and distinction of ID vs. OOD evaluation, as well as the subtleties of OOD evaluation with current benchmarks.
BibTeX
@inproceedings{setlur2021two,
  author = {Amrith Setlur and Oscar Li and Virginia Smith},
  title = {{Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=X4JfcKvvDOw},
}
Figure