On Episodes, Prototypical Networks, and Few-shot Learning

Steinar Laenen, Luca Bertinetto

TL;DR

This paper challenges the "train-as-you-test" dogma in few-shot learning, showing that the popular practice of episodic training is actually data-inefficient for metric-based models. By replacing complex episodic setups with a simpler, non-episodic Neighborhood Component Analysis (NCA) loss, the authors achieve better performance with fewer hyperparameters.

What Problem Does This Paper Address?

Few-shot learning (FSL) aims to train models that can recognize new categories from just one or five examples. For years, the gold standard for training these models has been episodic learning. The logic is intuitive: if you want a model to perform well on a 5-way, 1-shot task at test time, you should train it on thousands of similar 5-way, 1-shot "episodes" sampled from your training data.

However, the authors of this paper noticed a catch. Episodic training forces a strict division of every training batch into a "support set" (the references) and a "query set" (the examples to classify). This constraint means the model only learns from distances between queries and supports, completely ignoring distances between two query images or two support images. The authors ask: Is this artificial constraint actually helping, or is it just throwing away valuable training signal?

Methods

The researchers focus on two seminal metric-based FSL methods: Prototypical Networks (PNs) and Matching Networks (MNs). These models work by learning a deep embedding space where images of the same class are close together.

To test the necessity of episodes, they compare these methods against Neighborhood Component Analysis (NCA). NCA is a classic non-parametric method that treats an entire training batch as a single pool. In NCA, every image in the batch acts as a "query" that tries to find its "neighbors" among all other images in the same batch.

The key difference is illustrated in Figure 1: Difference in batch exploitation between episodic and non-episodic methods.

Difference in batch exploitation between episodic and non-episodic methods.

While episodic methods (left) only look at the dashed lines crossing the support/query divide, the non-episodic approach (right) exploits all possible pairwise distances within the batch.

Datasets

The authors validated their findings on three standard FSL benchmarks:

  • miniImageNet: A subset of ImageNet with 100 classes.
  • CIFAR-FS: A few-shot variant of the CIFAR-100 dataset.
  • tieredImageNet: A larger, more complex version of ImageNet with hierarchical splits to ensure training and test classes are semantically distinct.

In all cases, they used a ResNet-12 backbone to ensure a fair comparison with modern baselines.

Key Insights and Ideas

  • The "Pairs Lost" Problem: The authors mathematically prove that episodic training is data-inefficient. By splitting a batch of size $b$ into support and query sets, you discard a quadratic number of potential training pairs. As shown in Table 1 (page 2), ignoring episodic constraints can increase the number of informative gradients by over 150% in typical setups.
  • Hyperparameter Headaches: Episodic learning introduces "nuisance" hyperparameters: how many "ways" (classes) and "shots" (examples) should you use during training? The paper shows that PNs and MNs are highly sensitive to these choices, and the "best" training episode often doesn't match the test episode.
  • Simplicity Wins: NCA removes the need for these episodic hyperparameters entirely. You just need a standard batch size, much like regular supervised learning.
  • Episodic Training as Sub-sampling: Through ablation studies, the authors show that episodic training behaves almost exactly like taking a full batch and randomly throwing away a large percentage of the data pairs (see Figure 3).

Key Results

The results were consistent across all datasets: the non-episodic NCA approach outperformed the episodic versions of PNs and MNs.

  • Accuracy Boost: On miniImageNet (5-shot), the NCA approach achieved 78.27% accuracy, compared to 75.42% for the standard episodic Prototypical Network.
  • Robustness to Batch Size: As shown in Figure 2, NCA-trained embeddings perform better than all episodic configurations across various batch sizes.
1-shot and 5-shot accuracies on CIFAR-FS comparing NCA to various episodic PN configurations.

The authors also found that their "best" episodic configurations (found through extensive searching) were significantly better than those used in the original seminal papers, yet they still couldn't beat the simpler NCA baseline.

Limitations and Future Work

The authors acknowledge that their study is limited to non-parametric methods—those that don't update their internal weights at test time. It remains an open question whether these findings apply to "optimization-based" meta-learners like MAML, which perform explicit gradient steps during the few-shot task.

Additionally, while NCA is simpler, it does require a sufficiently large batch size to ensure that every image has at least one other image of the same class to "pull" towards. Future work could explore how to best balance class variety and examples-per-class in these non-episodic batches.

Ultimately, this paper serves as a reminder to the FSL community: sometimes the most "principled" approach (mimicking test conditions) is less effective than simply using all the data you have.

Summary updated

Abstract

Episodic learning is a popular practice among researchers and practitioners interested in few-shot learning. It consists of organising training in a series of learning problems (or episodes), each divided into a small training and validation subset to mimic the circumstances encountered during evaluation. But is this always necessary? In this paper, we investigate the usefulness of episodic learning in methods which use nonparametric approaches, such as nearest neighbours, at the level of the episode. For these methods, we not only show how the constraints imposed by episodic learning are not necessary, but that they in fact lead to a data-inefficient way of exploiting training batches. We conduct a wide range of ablative experiments with Matching and Prototypical Networks, two of the most popular methods that use nonparametric approaches at the level of the episode. Their "non-episodic" counterparts are considerably simpler, have less hyperparameters, and improve their performance in multiple few-shot classification datasets.
BibTeX
@inproceedings{laenen2021episodes,
  author = {Steinar Laenen and Luca Bertinetto},
  title = {{On Episodes, Prototypical Networks, and Few-shot Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=bJaZ8leI0QJ},
}
Figure