Abstract
Through sequential construction of posteriors on observing data online, Bayes' theorem provides a natural framework for continual learning. We develop Variational Auto-Regressive Gaussian Processes (VAR-GPs), a principled posterior updating mechanism to solve sequential tasks in continual learning. By relying on sparse inducing point approximations for scalable posteriors, we propose a novel auto-regressive variational distribution which reveals two fruitful connections to existing results in Bayesian inference: expectation propagation and orthogonal inducing points. Mean predictive entropy estimates show VAR-GPs prevent catastrophic forgetting, which is empirically supported by strong performance on modern continual learning benchmarks against competitive baselines. A thorough ablation study demonstrates the efficacy of our modeling choices.
Introduction
In Continual Learning (CL), models must learn a sequence of tasks over time without losing performance on earlier ones. Standard neural networks notoriously suffer from catastrophic forgetting—when they learn something new, their weights update and overwrite past knowledge.
Bayesian inference offers a natural theoretical solution: using the posterior of the old tasks as the prior for the new task. However, exact Bayesian inference is computationally intractable for complex models. While sparse Gaussian Processes (GPs) have been used for streaming data, they typically rely on a single, fixed-size set of "inducing points" (representative pseudo-data) shared across all time steps. This forces a harsh trade-off: the inducing points are either too rigid to learn new tasks or move too quickly and forget old ones.
The VAR-GP Framework
VAR-GPs fundamentally change how inducing points are handled across sequential tasks to resolve the forgetting trade-off:
- Task-Specific Inducing Points: Instead of recycling one set of inducing points, VAR-GPs allocate a separate set of inducing inputs and outputs for every new task.
- Auto-Regressive Variational Distribution: To connect the tasks, the authors propose an auto-regressive structure. The variational distribution for the current task's inducing points is explicitly conditioned on the inducing points of all previous tasks.
- Generalized Continual ELBO: This structure allows for a tractable Evidence Lower Bound (ELBO) that naturally balances maximizing the likelihood of new data while using Kullback-Leibler (KL) divergence to regularize against the posterior of past tasks.
Key Results and Insights
The authors evaluate VAR-GPs on several benchmarks, demonstrating that cross-task covariance is the key to memory. By explicitly modeling the covariance between old and new inducing points, VAR-GPs naturally bias the learning process to preserve high-density regions of past tasks.
MNIST Benchmarks
- Split MNIST: In a challenging single-head setup where task identity is unknown at test time, VAR-GPs achieved a final mean test accuracy of 90.57% across 5 tasks, significantly outperforming the strongest baseline (VCL with a coreset memory buffer) at 81.90%.
- Permuted MNIST: VAR-GPs achieved 97.20% accuracy after 10 tasks, compared to 95.06% for the best VCL variant.
Qualitative Uncertainty
Entropy maps show that VAR-GPs maintain low predictive uncertainty (high confidence) on old tasks even as new tasks are learned. Unlike many baselines, VAR-GPs achieve these results purely through the Bayesian posterior, without needing to store raw past data in a memory buffer.
Limitations and Future Work
Because VAR-GPs allocate a new set of inducing points for every task, the total number of points grows linearly. Since GP inference scales cubically, the computational cost scales as $O(T^3)$ where $T$ is the number of tasks. This currently limits the method to a moderate number of tasks. Future work may explore post-hoc information distillation to compress the growing number of inducing points or applying this framework to model-based reinforcement learning.
Summary updated