TL;DR
This paper introduces a novel Bayesian Online Meta-Learning (BOML) framework to address catastrophic forgetting in few-shot classification tasks. By combining Bayesian online learning with meta-learning, their approach effectively retains knowledge from previously encountered datasets, even when faced with distributional shifts, a significant limitation of existing meta-learning methods.
What Problem Does This Paper Address?
Traditional neural networks are prone to "catastrophic forgetting," where learning new information causes them to lose previously acquired knowledge. While this has been studied in large-scale supervised learning, it's a less-addressed challenge in few-shot classification. Few-shot learning aims to train models that can quickly adapt to new classes with very few examples. Meta-learning, or "learning to learn," has shown promise here, but standard meta-learning algorithms like MAML suffer from catastrophic forgetting when trained on sequential datasets with different underlying distributions. This means a model trained on one dataset's few-shot tasks might perform poorly on another's. The paper tackles this by developing a meta-learning framework that can learn from a sequence of datasets, each with its own distributional shift, without forgetting previous knowledge.
Methods
The core of the proposed solution is the Bayesian Online Meta-Learning (BOML) framework. It integrates two key ideas:
- Bayesian Online Learning (BOL): This approach treats model parameters as random variables with a probability distribution (posterior). As new data arrives, the posterior is updated recursively, naturally incorporating past knowledge. This contrasts with methods that explicitly store and re-use past data, which can be memory-intensive and computationally costly.
- Meta-Learning: This is used to find a good initial set of "meta-parameters" that can quickly adapt to new few-shot tasks.
BOML combines these by applying Bayesian online learning to the meta-parameters themselves. The framework uses Laplace approximation (BOMLA) or variational inference (BOMVI) to approximate the posterior of these meta-parameters. This allows the model to maintain a probabilistic representation of its learned knowledge, which is then used as a prior for learning from new, sequentially arriving datasets. A key technical contribution is the development of efficient Hessian and Fisher approximations for the meta-parameters in the Laplace approximation setting.
Datasets
The experiments were conducted on several standard few-shot learning benchmarks, including:
- Omniglot: A dataset of handwritten characters from various alphabets.
- miniQuickDraw: A dataset of drawings from the "Quick, Draw!" game.
- CIFAR-FS: A subset of the CIFAR-10 dataset adapted for few-shot learning.
- miniImageNet: A subset of the ImageNet dataset adapted for few-shot learning.
- VGG-Flowers: A dataset of flower images.
- Aircraft: A fine-grained dataset of aircraft models.
The experiments involved sequential training across these datasets to simulate the problem of learning from a stream of data with distributional shifts.
Key Insights and Ideas
- Catastrophic forgetting is a real problem for meta-learning: The paper empirically shows that standard MAML suffers from forgetting when faced with sequential, distributionally-shifted few-shot datasets.
- Bayesian online learning is a natural fit for sequential knowledge accumulation: By treating meta-parameters as distributions, BOL provides a principled way to update knowledge without explicit memory buffers.
- Applying BOL to meta-parameters is key: The novelty lies in using Bayesian online learning not just for model parameters, but for the meta-parameters that govern fast adaptation. This allows for continual meta-learning.
- Laplace approximation and variational inference offer practical solutions: These techniques provide tractable ways to approximate the complex posterior distributions of meta-parameters, enabling efficient training.
- The posterior covariance reveals learning dynamics: Visualizing the approximate posterior covariance shows how the model's uncertainty about its meta-parameters changes as it learns from new datasets, offering insights into the adaptation process.
Key Results
- BOMLA and BOMVI effectively prevent catastrophic forgetting: In sequential training experiments (Triathlon and Pentathlon sequences), the proposed BOML methods significantly outperform baselines like Sequential MAML, retaining high accuracy on previously learned datasets even after training on new ones.
- BOMLA shows strong performance, especially with appropriate regularization: The BOMLA method, particularly with a tuned posterior regularizer ($\lambda$), demonstrates a good balance between learning new datasets and retaining old ones.
- BOML can continually learn from stationary task distributions: In the Omniglot experiment, where tasks originate from the same distribution, BOML also shows the ability to accumulate few-shot classification ability over time.
- BOMLA generally outperforms BOMVI: This is attributed to BOMLA's more expressive Gaussian approximate posterior (block-diagonal) compared to BOMVI's mean-field approximation.
Limitations and Future Work
- Posterior approximation trade-offs: The paper acknowledges that there's a trade-off between retaining performance on old datasets and learning well on new ones due to posterior approximation errors. Tuning the regularization parameter $\lambda$ in BOMLA is crucial for balancing this.
- Computational cost: While BOML is more efficient than methods that store all past data, the computational cost can still be significant, especially for the Hessian approximation in BOMLA.
- Future work: The authors suggest exploring more advanced variational inference techniques for BOMVI to improve its performance and potentially reduce sampling variance. Further investigation into the theoretical properties of the approximate posterior covariance and its impact on forgetting is also a promising direction.
Summary updated