Abstract
In continual learning, knowledge must be preserved and re-used between tasks, maintaining good transfer to future tasks and minimizing forgetting of previously learned ones. While several practical algorithms have been devised for this setting, there have been few theoretical works aiming to quantify and bound the degree of Forgetting in general settings. We provide both data-dependent and oracle upper bounds that apply regardless of model and algorithm choice, as well as bounds for Gibbs posteriors. We derive an algorithm inspired by our bounds and demonstrate empirically that our approach yields improved forward and backward transfer.
The Theoretical Gap in Continual Learning
Continual Learning (CL) involves an agent learning a sequence of tasks sequentially. A primary challenge is Catastrophic Forgetting, where learning new information causes the model to lose performance on previously mastered tasks. While numerous heuristics—such as experience replay or weight regularization—have been developed to mitigate this, a rigorous theoretical understanding has remained elusive.
Existing theoretical analyses are often constrained to specific regimes, such as linear regression or the Neural Tangent Kernel (NTK) limit. These constraints limit their applicability to the deep, non-linear architectures used in practice. This paper addresses this gap by introducing a model-agnostic framework based on PAC-Bayes theory.
As shown in the table above, while previous works were limited by architecture or task construction, the proposed PAC-Bayes approach provides bounds that are applicable to any architecture and a wide range of task settings.
A PAC-Bayesian Framework for Forgetting
The authors utilize the PAC-Bayes framework to derive "probably approximately correct" bounds on generalization error. This approach is particularly suited for CL because it naturally handles the transition from a prior distribution (knowledge from previous tasks) to a posterior distribution (knowledge updated with the current task).
Data-Dependent Bounds
The core contribution is a bound for Backward Transfer (BWT), which quantifies performance degradation on old tasks. This bound is composed of three intuitive terms:
- Empirical Loss: The model's performance on the current task.
- KL-Divergence: A measure of how much the model parameters have shifted from the previous task's state.
- Disagreement Term: A term capturing the "loss covariance" or landscape similarity between the old and new tasks.
Oracle Bounds and Gibbs Posteriors
For idealized learners using a Gibbs posterior, the authors derive "oracle" bounds. These represent the theoretical limit of performance if the underlying data distributions were known, providing a benchmark for practical algorithms.
From Theory to Practice: PAC-Bayes CL
The theoretical bounds directly inspire a new optimization objective. The proposed PAC-Bayes Continual Learning algorithm (implemented via Variational Inference, or VI) treats the posterior from task $t-1$ as the prior for task $t$. The objective function explicitly balances minimizing the current task's loss with a KL-regularization term that prevents the model from moving too far from the previous solution, thereby preserving old knowledge.
Empirical Results and Bound Tightness
The framework was evaluated on synthetic Gaussian data with varying task shifts (Similar, Gradual, and Orthogonal) and standard benchmarks like Permuted-MNIST and Split-CIFAR10.
A key finding is that the derived PAC-Bayes bounds are remarkably tight. Unlike many theoretical bounds that provide non-informative (vacuous) values, these bounds closely track the actual forgetting observed during training. In synthetic experiments, the VI algorithm often outperformed Elastic Weight Consolidation (EWC), particularly in maintaining backward transfer as the number of tasks increased.
Conclusion and Future Directions
This work provides the first general, model-agnostic theoretical foundation for understanding forgetting in continual learning. By demonstrating that PAC-Bayes bounds can be both informative and actionable, it opens new avenues for theoretically-grounded CL research.
Future work includes extending these proofs to non-independent task sequences (curriculum learning) and addressing the computational scaling of KL-divergence estimation for very large-scale neural networks.
Summary updated