Continual Learning from the Perspective of Compression

Xu He, Min LinarXiv 2020 · arXiv

Introduction

Neural networks are notoriously susceptible to catastrophic forgetting: the abrupt loss of knowledge about previous tasks upon learning new ones. Traditionally, this phenomenon is measured through task-specific metrics like classification accuracy or regression error. However, these metrics often fail to capture the nuances of what information is actually lost. This paper redefines catastrophic forgetting through the lens of information theory, specifically utilizing the Minimum Description Length (MDL) principle. By framing learning as a data compression problem, the authors provide a principled way to quantify information loss and bridge the gap between popular continual learning (CL) paradigms.

Redefining Forgetting as Information Loss

The core contribution of this work is the formalization of forgetting as an increase in "codelength"—the number of bits required to describe past data $x$ when model parameters transition from $\theta_i$ to $\theta_j$. If a model requires more bits to describe old data after learning a new task, it has effectively "forgotten" the regularities of that data. This perspective allows for a unified evaluation of diverse model types, including generative models like Variational Autoencoders (VAEs), where traditional accuracy metrics are inapplicable.

Theoretical Framework and the ML Mixture Method

The authors demonstrate that existing CL methods are actually approximations of classic coding strategies:

  • Variational Continual Learning (VCL) serves as an approximation of Bayesian Mixture Coding.
  • Generative Replay approximates Maximum Likelihood (ML) Plug-in Coding.

Building on these insights, the paper proposes the ML Mixture method. This hierarchical approach learns hyperparameters $\phi$ (representing weight means and variances) to maximize data likelihood. By projecting data into a "local posterior" within the parameter space, the ML Mixture method better models the distribution of weights across tasks, addressing the limitations of standard Bayesian approaches.

Experimental Evaluation

The methods were evaluated using a class-incremental setup on the MNIST dataset. In this benchmark, the model encounters digits sequentially (0, then 1, etc.) and must retain knowledge of previous digits without re-accessing their raw data. The primary metrics used were cumulative average forgetting and prequential codelength, both measured in bits per dimension (bpd).

Results and Discussion

The experimental results highlight a significant performance gap between traditional Bayesian methods and the proposed approach.

Cumulative average forgetting across class IDs.

As shown in the figure above, the theoretical "ideal" methods (ML Plug-in and Bayesian Mixture), which have access to all past data, exhibit near-zero forgetting. The proposed ML Mixture method achieves significantly lower forgetting than standard VCL, which suffers from high variability and a rapid increase in bits required for older tasks.

Comparison of prequential codelength across different continual learning methods.

The overall compression efficiency, summarized in the bar chart above, further validates the superiority of the ML Mixture. The authors identify two primary reasons for VCL's poor performance:

  1. The Prior Penalty: In Bayesian methods, the KL-divergence term acts as a regularization that adds bits to the description length. A suboptimal prior leads to "wasted" bits and poorer retention.
  2. The Variational Gap: VCL typically uses simple diagonal Gaussian distributions to approximate the complex, multimodal posterior of a neural network. This approximation error is a major hurdle for Bayesian CL methods.

Conclusion and Future Work

By treating comprehension as compression, this work provides a robust framework for evaluating and improving continual learning. While the ML Mixture method shows great promise, the authors note that its current "greedy" optimization can lead to a bias toward early tasks. Future research will focus on developing objectives with bounded approximation errors and exploring how minimal generative replay might be used to further adapt the model's prior to later tasks.

Summary updated

BibTeX
@misc{he2020continual,
  author = {Xu He and Min Lin},
  title = {{Continual Learning from the Perspective of Compression}},
  year = {2020},
  eprint = {2006.15078},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.15078},
}
Figure