Variational Auto-Regressive Gaussian Processes for Continual Learning

Sanyam Kapoor, Theofanis Karaletsos, Thang D. Bui

Abstract

Through sequential construction of posteriors on observing data online, Bayes' theorem provides a natural framework for continual learning. We develop Variational Auto-Regressive Gaussian Processes (VAR-GPs), a principled posterior updating mechanism to solve sequential tasks in continual learning. By relying on sparse inducing point approximations for scalable posteriors, we propose a novel auto-regressive variational distribution which reveals two fruitful connections to existing results in Bayesian inference: expectation propagation and orthogonal inducing points. Mean predictive entropy estimates show VAR-GPs prevent catastrophic forgetting, which is empirically supported by strong performance on modern continual learning benchmarks against competitive baselines. A thorough ablation study demonstrates the efficacy of our modeling choices.

Introduction

In Continual Learning (CL), models must learn a sequence of tasks over time without losing performance on earlier ones. Standard neural networks notoriously suffer from catastrophic forgetting—when they learn something new, their weights update and overwrite past knowledge.

Bayesian inference offers a natural theoretical solution: using the posterior of the old tasks as the prior for the new task. However, exact Bayesian inference is computationally intractable for complex models. While sparse Gaussian Processes (GPs) have been used for streaming data, they typically rely on a single, fixed-size set of "inducing points" (representative pseudo-data) shared across all time steps. This forces a harsh trade-off: the inducing points are either too rigid to learn new tasks or move too quickly and forget old ones.

The VAR-GP Framework

VAR-GPs fundamentally change how inducing points are handled across sequential tasks to resolve the forgetting trade-off:

  • Task-Specific Inducing Points: Instead of recycling one set of inducing points, VAR-GPs allocate a separate set of inducing inputs and outputs for every new task.
  • Auto-Regressive Variational Distribution: To connect the tasks, the authors propose an auto-regressive structure. The variational distribution for the current task's inducing points is explicitly conditioned on the inducing points of all previous tasks.
  • Generalized Continual ELBO: This structure allows for a tractable Evidence Lower Bound (ELBO) that naturally balances maximizing the likelihood of new data while using Kullback-Leibler (KL) divergence to regularize against the posterior of past tasks.
Qualitative comparison of predictive uncertainty on synthetic 2D data. VAR-GP (top) maintains well-defined regions for old tasks, whereas the baseline (bottom) shows significant degradation and overconfidence.

Key Results and Insights

The authors evaluate VAR-GPs on several benchmarks, demonstrating that cross-task covariance is the key to memory. By explicitly modeling the covariance between old and new inducing points, VAR-GPs naturally bias the learning process to preserve high-density regions of past tasks.

MNIST Benchmarks

  • Split MNIST: In a challenging single-head setup where task identity is unknown at test time, VAR-GPs achieved a final mean test accuracy of 90.57% across 5 tasks, significantly outperforming the strongest baseline (VCL with a coreset memory buffer) at 81.90%.
  • Permuted MNIST: VAR-GPs achieved 97.20% accuracy after 10 tasks, compared to 95.06% for the best VCL variant.
Performance on Permuted MNIST. The matrices show accuracy on all seen tasks (x-axis) after training on sequential tasks (y-axis). VAR-GP demonstrates superior retention compared to VCL.

Qualitative Uncertainty

Entropy maps show that VAR-GPs maintain low predictive uncertainty (high confidence) on old tasks even as new tasks are learned. Unlike many baselines, VAR-GPs achieve these results purely through the Bayesian posterior, without needing to store raw past data in a memory buffer.

Limitations and Future Work

Because VAR-GPs allocate a new set of inducing points for every task, the total number of points grows linearly. Since GP inference scales cubically, the computational cost scales as $O(T^3)$ where $T$ is the number of tasks. This currently limits the method to a moderate number of tasks. Future work may explore post-hoc information distillation to compress the growing number of inducing points or applying this framework to model-based reinforcement learning.

Summary updated

Abstract

Through sequential construction of posteriors on observing data online, Bayes' theorem provides a natural framework for continual learning. We develop Variational Auto-Regressive Gaussian Processes (VAR-GPs), a principled posterior updating mechanism to solve sequential tasks in continual learning. By relying on sparse inducing point approximations for scalable posteriors, we propose a novel auto-regressive variational distribution which reveals two fruitful connections to existing results in Bayesian inference, expectation propagation and orthogonal inducing points. Mean predictive entropy estimates show VAR-GPs prevent catastrophic forgetting, which is empirically supported by strong performance on modern continual learning benchmarks against competitive baselines. A thorough ablation study demonstrates the efficacy of our modeling choices.
BibTeX
@inproceedings{kapoor2021variational,
  author = {Sanyam Kapoor and Theofanis Karaletsos and Thang D. Bui},
  title = {{Variational Auto-Regressive Gaussian Processes for Continual Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=rAgWG62Hjb5},
}
Figure