Variational Auto-Regressive Gaussian Processes for Continual Learning

Sanyam Kapoor, Theofanis Karaletsos, Thang D. Bui

The Challenge of Continual Learning

In the real world, data often arrives in a continuous stream rather than all at once. Ideally, an AI system should be able to learn from new data without forgetting what it already knows—a paradigm known as Continual Learning (CL).

However, standard machine learning models, particularly neural networks, suffer from catastrophic forgetting. When trained on a new task, they tend to overwrite the weights used for previous tasks, causing performance on old data to plummet. Researchers face a "stability-plasticity" dilemma: how do we make a model plastic enough to learn new concepts, but rigid enough to retain old memories? While Bayesian methods provide a natural mathematical framework for this (using today's posterior as tomorrow's prior), scaling these methods to complex tasks without losing information remains a significant challenge.

Introducing VAR-GPs

To address this, researchers have developed Variational Auto-Regressive Gaussian Processes (VAR-GPs), a novel Bayesian framework that effectively combats catastrophic forgetting. By using a structured, auto-regressive variational distribution and task-specific inducing points, the model preserves knowledge from past tasks while adapting to new ones, significantly outperforming existing Bayesian neural network baselines.

The authors build their solution on Sparse Variational Gaussian Processes (SVGPs). Standard SVGPs use a small set of "inducing points" to approximate the full dataset, making Gaussian Processes computationally feasible. The core innovation in VAR-GPs is twofold:

  1. Task-Specific Inducing Points: Instead of trying to update a single set of inducing points for every new task (which can lead to the points "drifting" away from old data), the authors introduce a new, separate set of inducing points for every task. By adding new inducing points rather than overwriting old ones, the model physically "allocates" memory to different parts of the input space.
  2. Auto-Regressive Variational Distribution: They propose a novel way to parameterize the uncertainty (the variational distribution). By making the current task's inducing outputs depend on all previous ones in an auto-regressive chain, the model explicitly accounts for cross-task correlations.

This structure reveals deep theoretical links to other established methods like Expectation Propagation and Orthogonal Inducing Points, providing a principled foundation for its success.

Datasets and Evaluation

The model was evaluated on three key benchmarks to test its robustness and adaptability:

  • Synthetic 2D Dataset: A four-way classification task used for qualitative visualization of decision boundaries.
Synthetic 2D dataset used for qualitative evaluation of decision boundaries.
  • Split MNIST: The classic digit dataset split into five sequential tasks (0/1, then 2/3, etc.). Crucially, the authors use a "single-head" setup, meaning the model isn't told which task it's performing during testing—a much harder and more realistic scenario than the "multi-head" setup often seen in literature.
  • Permuted MNIST: Ten sequential tasks where each task is a different random permutation of MNIST pixels, testing the model's ability to handle significant distributional shifts.

Key Insights and Results

VAR-GPs demonstrated superior performance against competitive baselines like Variational Continual Learning (VCL):

  • Split MNIST: VAR-GP achieved a final average accuracy of 90.57%, dwarfing the best VCL variant (with coresets) at 81.90%. Without coresets, VCL collapsed to ~20%.
  • Permuted MNIST: VAR-GP reached 97.20% accuracy after 10 tasks, outperforming VCL variants which peaked around 95%.

Qualitatively, VAR-GPs preserve the "predictive density surface" of old tasks. While other models' predictions become distorted or overconfident, VAR-GPs keep the decision boundaries intact even after learning new classes.

Decision boundaries for VAR-GP vs VCL. VAR-GP (top) maintains stable predictive density for old tasks, whereas VCL (bottom) suffers from structural instability.

Two critical insights emerged from the study:

  1. Hyperparameter Uncertainty Matters: Treating kernel hyperparameters (like lengthscales) as point estimates leads to failure. The model needs to maintain a distribution (uncertainty) over these parameters to successfully bridge the gap between tasks.
  2. Predictive Entropy as a Shield: VAR-GPs maintain high uncertainty (entropy) in regions of the input space they haven't seen yet, while keeping entropy low for seen tasks. This prevents the model from being "overconfident" and making wild guesses on old data after learning something new.

Limitations and Future Work

The primary drawback of the current VAR-GP approach is scalability. Because the number of inducing points grows linearly with the number of tasks, the computational complexity for inference grows cubically. This limits the model to a moderate number of tasks.

To address this, future work could explore Information Distillation (compressing or pruning old inducing points that are no longer critical) and Deep Kernel Learning (combining the principled Bayesian framework of VAR-GPs with the representational power of deep neural networks to handle high-resolution images). Applying this framework to Reinforcement Learning agents that must learn new skills sequentially in changing environments also presents an exciting frontier.

Summary updated

Abstract

Through sequential construction of posteriors on observing data online, Bayes’ theorem provides a natural framework for continual learning. We develop Variational Auto-Regressive Gaussian Processes...
BibTeX
@inproceedings{kapoor2021variational,
  author = {Sanyam Kapoor and Theofanis Karaletsos and Thang D. Bui},
  title = {{Variational Auto-Regressive Gaussian Processes for Continual Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=rAgWG62Hjb5},
}
Figure