Generalized Variational Continual Learning

Noel Loo, Siddharth Swaroop, Richard E. Turner

Abstract

Continual learning requires models to learn new tasks sequentially without forgetting previous ones. Current probabilistic approaches, such as Variational Continual Learning (VCL) and Online Elastic Weight Consolidation (Online EWC), have largely existed as separate paradigms. We introduce Generalized Variational Continual Learning (GVCL), a framework that unifies these methods through likelihood-tempering. Furthermore, we address the "overpruning" pathology inherent in variational methods by incorporating task-specific Feature Linear Modulation (FiLM) layers. Our results demonstrate that GVCL with FiLM (GVCL-F) significantly outperforms existing baselines in small-data regimes and provides superior uncertainty calibration on larger benchmarks.

The Challenge of Continual Learning

Neural networks typically suffer from catastrophic forgetting: when trained on a new task, the weights required for previous tasks are overwritten. Regularization-based solutions often fall into two categories: Laplace-based methods like Online EWC, which use point estimates, and Variational methods like VCL, which treat weights as distributions.

While VCL is theoretically robust, it suffers from a "pathological" pruning effect. In an effort to minimize complexity (the KL-divergence term in the ELBO), the model aggressively shuts down neurons. Once a neuron's bias is set to a large negative value to "silence" it, it is effectively locked out of future learning, wasting the network's total capacity.

Methodology: GVCL and FiLM Layers

1. Unifying VCL and EWC

We show that by applying "likelihood-tempering"—introducing a hyperparameter $\beta$ to the Evidence Lower Bound (ELBO)—VCL recovers Online EWC as a limiting case when $\beta \to 0$. This allows for a unified algorithm, GVCL, which can interpolate between the global curvature of VCL and the local curvature of EWC to find an optimal balance for a given dataset.

2. Solving Overpruning with FiLM

To mitigate overpruning, we introduce task-specific FiLM layers. These layers apply a simple linear transformation (scaling and shifting) to the activations of the main network. Crucially, because these parameters are task-specific, they allow the model to "silence" a neuron for Task A while keeping it available to be "re-activated" for Task B. This architectural change prevents the permanent loss of model capacity seen in standard variational approaches.

Comparison of filter usage between standard GVCL and GVCL with FiLM layers. FiLM allows the model to reuse and re-activate filters across different tasks, whereas standard GVCL permanently prunes them.

Experimental Results

Small-Data Dominance: The CHASY Benchmark

We evaluated GVCL-F on CHASY, a benchmark based on LaTeX symbols designed to test transfer in small-data regimes (16 samples per class). In these few-shot scenarios, GVCL-F significantly outperformed strong baselines like HAT and PathNet. For example, on Easy-CHASY, GVCL-F achieved 90.9% accuracy, a substantial lead over HAT's 82.6%.

Average accuracy on the Easy-CHASY benchmark. GVCL-F (red) maintains a significant performance lead over standard GVCL (green) and HAT (blue) as the number of tasks increases.

Calibration and Large-Scale Performance

On larger benchmarks like Mixed Vision (a sequence of 8 distinct datasets), GVCL-F matched the accuracy of state-of-the-art methods like HAT while providing vastly superior Expected Calibration Error (ECE). GVCL-F achieved an ECE of 0.3%, compared to 1.7% for HAT. This indicates that the model's confidence scores are highly reliable, a critical requirement for real-world deployment.

Conclusion

GVCL provides a theoretical bridge between two major strands of continual learning research. By combining this unified framework with FiLM layers, we solve the overpruning problem that has previously limited variational methods. The resulting approach is not only more accurate in data-scarce environments but also provides the well-calibrated uncertainty estimates necessary for robust Bayesian inference.

Summary updated

Abstract

Continual learning deals with training models on new tasks and datasets in an online fashion. One strand of research has used probabilistic regularization for continual learning, with two of the main approaches in this vein being Online Elastic Weight Consolidation (Online EWC) and Variational Continual Learning (VCL). VCL employs variational inference, which in other settings has been improved empirically by applying likelihood-tempering. We show that applying this modification to VCL recovers Online EWC as a limiting case, allowing for interpolation between the two approaches. We term the general algorithm Generalized VCL (GVCL). In order to mitigate the observed overpruning effect of VI, we take inspiration from a common multi-task architecture, neural networks with task-specific FiLM layers, and find that this addition leads to significant performance gains, specifically for variational methods. In the small-data regime, GVCL strongly outperforms existing baselines. In larger datasets, GVCL with FiLM layers outperforms or is competitive with existing baselines in terms of accuracy, whilst also providing significantly better calibration.
BibTeX
@inproceedings{loo2021generalized,
  author = {Noel Loo and Siddharth Swaroop and Richard E. Turner},
  title = {{Generalized Variational Continual Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=_IM-AfFhna9},
}
Figure