On Episodes, Prototypical Networks, and Few-Shot Learning

Steinar Laenen, Luca Bertinetto

TL;DR

This paper challenges the widely held belief that "episodic learning"—training on small, simulated tasks—is necessary for effective few-shot learning. The authors demonstrate that for metric-based methods like Prototypical Networks, episodic constraints are actually data-inefficient. By switching to a simpler, non-episodic approach called Neighborhood Component Analysis (NCA), they achieve better performance with fewer hyperparameters.

What Problem Does This Paper Address?

In Few-Shot Learning (FSL), the goal is to classify new objects after seeing only one or a few examples (the "shots"). For years, the dominant paradigm has been episodic learning, based on the principle that "train and test conditions should match." During training, researchers create "episodes" consisting of a tiny support set (to learn from) and a query set (to test on), mimicking the final evaluation.

However, the authors noticed that while episodic learning is intuitive, it might be unnecessarily restrictive. By forcing a batch of data into the rigid structure of an episode, are we throwing away valuable information? This paper investigates whether the "match your training to your testing" dogma actually holds up for metric-based FSL methods.

Methods

The authors compare three main loss functions:

  1. Prototypical Networks (PNs): An episodic method that represents each class by the mean vector (prototype) of its support examples.
  2. Matching Networks (MNs): An episodic method that compares query images to every individual support example using a softmax over distances.
  3. Neighborhood Component Analysis (NCA): A non-episodic method. Unlike PNs or MNs, NCA doesn't distinguish between "support" and "query" sets during training. Instead, it looks at a standard mini-batch and calculates the probability of each image being correctly classified based on its proximity to all other images of the same class in that batch.

The key technical contribution is showing that NCA is essentially a "limitless" version of Matching Networks that exploits every possible pairwise distance within a training batch, rather than just the subset allowed by the episodic structure.

The Data Efficiency Gap

The authors prove that episodic learning is mathematically "wasteful." In a typical batch, the episodic constraint ignores a massive number of potential training signals.

Comparison of pairwise distances computed in episodic vs. non-episodic learning.

As illustrated above, the episodic structure (left) artificially restricts comparisons to only those between the query (Q) and support (S) sets. In contrast, the non-episodic approach (right) leverages all possible connections within the batch (B). For a standard setup (20 ways, 5 shots, 15 queries), ignoring episodic constraints increases the number of available positive and negative pairs by more than 150%.

Key Insights and Ideas

  • Episodes as Random Sub-sampling: Through ablation studies, the authors show that training on episodes is empirically equivalent to taking a full batch of data and randomly discarding a large fraction of the pairwise distances. There is no "magic" in the episodic structure; it just acts as a noisy sub-sampler.
  • Hyperparameter Simplification: Episodic learning requires tuning the number of "ways" ($w$), "shots" ($n$), and "queries" ($m$) used during training. NCA removes these, leaving only the standard batch size to tune, which makes the training process significantly more robust and easier to implement.
  • Evaluation Matters: The paper highlights that "Nearest Centroid" evaluation (assigning a test image to the class with the closest average embedding) is remarkably effective, often outperforming more complex soft-assignment methods.

Key Results

The "non-episodic" NCA approach consistently outperformed its episodic counterparts (PNs and MNs) across standard few-shot classification benchmarks (miniImageNet, CIFAR-FS, and tieredImageNet) using a ResNet-12 backbone.

Accuracy comparison between NCA and episodic methods across varying batch sizes.
  • miniImageNet: NCA achieved 62.55% (1-shot) and 78.27% (5-shot), outperforming the original episodic Prototypical Networks implementation by a wide margin (~3-5%).
  • CIFAR-FS: NCA reached 72.49% (1-shot) and 85.15% (5-shot).
  • Competitive with SOTA: Despite its extreme simplicity, the vanilla NCA approach performed as well as, or better than, many complex meta-learning algorithms (like MetaOptNet or TADAM) that involve much more sophisticated machinery.

Limitations and Future Work

  • Scope of Methods: The study focuses specifically on metric-based (nonparametric) methods. It does not necessarily prove that episodic learning is useless for optimization-based methods like MAML, where the "inner loop" adaptation might still benefit from the episodic structure.
  • Batch Composition: While NCA is less sensitive than episodic methods, it still relies on having enough "positive" pairs (images of the same class) within a batch. In datasets with a massive number of classes and few images per class, batch sampling strategy remains important.
  • Future Directions: The authors suggest further research into the specific roles of positive vs. negative distances in the loss function, as their analysis showed that the balance between these two is a primary driver of performance.

Summary updated

Abstract

Episodic learning is a popular practice among researchers and practitioners interested in few-shot learning. It consists of organising training in a series of learning problems (or episodes), each divided into a small training and validation subset to mimic the circumstances encountered during evaluation. But is this always necessary? In this paper, we investigate the usefulness of episodic learning in methods which use nonparametric approaches, such as nearest neighbours, at the level of the episode. For these methods, we not only show how the constraints imposed by episodic learning are not necessary, but that they in fact lead to a data-inefficient way of exploiting training batches. We conduct a wide range of ablative experiments with Matching and Prototypical Networks, two of the most popular methods that use nonparametric approaches at the level of the episode. Their "non-episodic'' counterparts are considerably simpler, have less hyperparameters, and improve their performance in multiple few-shot classification datasets.
BibTeX
@inproceedings{laenen2021episodes,
  author = {Steinar Laenen and Luca Bertinetto},
  title = {{On Episodes, Prototypical Networks, and Few-Shot Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=bJaZ8leI0QJ},
}
Figure