Addressing Catastrophic Forgetting in Few-Shot Problems

Pau Ching Yap, Hippolyt Ritter, David Barber

Introduction

Neural networks are notoriously prone to catastrophic forgetting—when trained on a new dataset, they tend to overwrite the weights used for previous datasets, effectively erasing prior knowledge. While this is a well-documented issue in standard supervised learning, it also severely impacts meta-learning (learning to learn).

Popular algorithms like MAML (Model-Agnostic Meta-Learning) are typically trained on a stationary distribution of tasks. However, in real-world scenarios, data often arrives in a continuous stream. If you meta-train MAML on a sequence of diverse domains—for instance, first learning to recognize handwritten Latin characters and then moving to natural images of animals—MAML will eventually lose its ability to quickly adapt to the character dataset.

To address this, researchers have introduced Bayesian Online Meta-Learning (BOML), a framework designed to prevent few-shot learning models from forgetting how to learn old tasks when trained on new ones. By treating meta-parameters as a probability distribution and updating them recursively using Bayesian principles, BOML enables models to successfully navigate a sequence of diverse datasets without losing their previous expertise.

The BOML Framework

The authors propose merging Bayesian Online Learning (BOL) with meta-learning. Instead of finding a single point estimate for meta-parameters (the initial weights), the BOML framework maintains a posterior distribution over them.

BOML Process Flow

The researchers introduce two specific implementations of this framework:

  1. BOMLA (BOML with Laplace Approximation): This method uses a Gaussian approximation of the posterior. To make this computationally feasible for neural networks, the authors derive a block-diagonal Kronecker-factored Hessian approximation. This allows the model to estimate parameter uncertainty without the massive overhead of computing a full Hessian matrix.
  2. BOMVI (BOML with Variational Inference): This approach adapts Variational Continual Learning (VCL) to the meta-learning setting, minimizing the KL-divergence between the current posterior and the previous one.

The Key Innovation: Unlike previous work that applied Laplace approximations to the task-specific level (the inner loop of meta-learning), BOML applies it to the meta-parameter level (the outer loop). This ensures that the "initialization" of the model stays in a region of the parameter space that works well for all datasets encountered so far.

Experimental Setup

The researchers evaluated their methods on sequential few-shot classification "marathons":

  • Triathlon: A sequence of three datasets: Omniglot $\rightarrow$ miniQuickDraw $\rightarrow$ CIFAR-FS.
  • Pentathlon: A more challenging five-dataset sequence: Omniglot $\rightarrow$ CIFAR-FS $\rightarrow$ miniImageNet $\rightarrow$ VGG-Flowers $\rightarrow$ Aircraft.
  • Stationary Omniglot: A sequence of tasks all drawn from the same Omniglot distribution to test if the framework helps in standard, non-shifting environments.

Key Results

Prevention of Forgetting

In the rigorous Pentathlon experiment, BOMLA significantly outperformed standard Sequential MAML.

Pentathlon Meta-Evaluation Results

As shown in the results, MAML's accuracy on early datasets (like Omniglot) plummeted as it learned new domains. In contrast, BOMLA maintained high performance across the entire sequence. While baselines like "Train-On-Everything" (TOE) achieve this by keeping all old data (explicit memory), BOML achieves it through a "probabilistic memory" (implicit memory), making its algorithmic complexity independent of the number of datasets seen.

BOMLA vs. BOMVI

Across the experiments, BOMLA generally performed better than BOMVI. The authors attribute this to the block-diagonal Kronecker-factored covariance in BOMLA being a richer approximation than the mean-field (diagonal) approximation typically used in Variational Inference.

Forward Transfer

On the stationary Omniglot task, the BOML framework demonstrated "forward transfer." The model actually improved at learning new tasks over time by accumulating knowledge from previous ones, whereas standard baselines remained stagnant.

Limitations and Future Work

While BOML presents a strong step forward, it comes with a few challenges:

  • Hyperparameter Sensitivity: BOMLA introduces a posterior-regularizing hyperparameter ($\lambda$) that requires careful tuning to balance the trade-off between remembering old tasks and learning new ones.
  • Sampling Variance: BOMVI suffered from high estimator variance due to the Monte Carlo sampling required for the variational objective.
  • Scalability: While Kronecker factors help, calculating second-order information for massive architectures (like large Transformers) remains computationally demanding. Future work will need to explore even more efficient curvature approximations to scale these Bayesian methods to the largest modern models.

Summary updated

Abstract

Neural networks are known to suffer from catastrophic forgetting when trained on sequential datasets. While there have been numerous attempts to solve this problem in large-scale supervised classification, little has been done to overcome catastrophic forgetting in few-shot classification problems. We demonstrate that the popular gradient-based model-agnostic meta-learning algorithm (MAML) indeed suffers from catastrophic forgetting and introduce a Bayesian online meta-learning framework that tackles this problem. Our framework utilises Bayesian online learning and meta-learning along with Laplace approximation and variational inference to overcome catastrophic forgetting in few-shot classification problems. The experimental evaluations demonstrate that our framework can effectively achieve this goal in comparison with various baselines. As an additional utility, we also demonstrate empirically that our framework is capable of meta-learning on sequentially arriving few-shot tasks from a stationary task distribution.
BibTeX
@inproceedings{yap2021addressing,
  author = {Pau Ching Yap and Hippolyt Ritter and David Barber},
  title = {{Addressing Catastrophic Forgetting in Few-Shot Problems}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=xWBXwZVo0W},
}
Figure