Abstract
Online continual learning faces the challenge of representation shift, where updating an encoder to learn new data causes the feature mappings of old data to drift, leading to catastrophic forgetting. We propose DeepCCG (Deep Class-Conditional Gaussians), a method that replaces standard softmax classifiers with a Bayesian class-conditional Gaussian model. This allows for instantaneous realignment of the classifier to shifting representations in a single step. Combined with a Log Conditional Marginal Likelihood loss and information-theoretic sample selection, DeepCCG significantly reduces forgetting and outperforms state-of-the-art methods on standard benchmarks like CIFAR and MiniImageNet.
Introduction: The Challenge of Representation Shift
In online continual learning, models typically consist of an encoder that creates a representation of the data and a classifier that assigns labels. A major hurdle is representation shift: as the encoder updates to learn new information, the "location" of old data in the feature space moves.
Standard classifiers (like a linear layer followed by a softmax) adapt to these shifts slowly via gradient descent. This creates a period of "misalignment" where the classifier is looking for old classes in the wrong place. This noise during learning often forces the encoder to "forget" old information unnecessarily to satisfy an outdated classifier.
Methods: Deep Class-Conditional Gaussians
DeepCCG introduces three key innovations to maintain alignment between the feature extractor and the classifier:
1. Bayesian Class-Conditional Gaussian Classifier
Instead of a standard neural network head, DeepCCG models each class as a Gaussian distribution in the representation space. Because this is a tractable Bayesian model, the classifier's decision boundaries (the posterior) can be recomputed in a single step whenever the representation shifts. This "instantaneous" adaptation prevents the encoder from receiving noisy signals.
2. Log Conditional Marginal Likelihood (LCML) Loss
To train the encoder, the authors use a loss function that maximizes the probability of labels given the representation, conditioned on stored memory. This ensures the representation is optimized specifically to be well-separated for the Gaussian classifier.
3. Information-Theoretic Sample Selection
To maintain a fixed-size memory buffer, DeepCCG selects examples that minimize the KL divergence between the "true" posterior (based on all seen data) and the posterior induced by the subset in memory. This ensures the stored samples are the most representative of the class means.
Experimental Results
The method was evaluated on three standard computer vision benchmarks: CIFAR-10, CIFAR-100, and MiniImageNet. DeepCCG consistently outperformed a wide spectrum of competitive baselines, including ER-reservoir, DER++, and iCaRL.
- Accuracy Boost: On the CIFAR-100 Class-Incremental benchmark, DeepCCG achieved a mean accuracy of 18.00%, significantly higher than the next best method, ER-ACE (13.51%).
- Stability: Analysis showed that for any given amount of shift in the feature space, DeepCCG’s accuracy dropped significantly less than other methods, proving it is more robust to the internal "moving targets" of neural network training.
- Efficiency: Because the classifier update is a single-step calculation rather than an iterative process, it maintains high performance without adding significant computational overhead during the update phase.
Limitations and Future Work
While effective, DeepCCG assumes no interaction between the mean parameters of different classes, which could lead to clashes in dense class-incremental settings. Additionally, the model requires a "critical mass" of samples to accurately estimate Gaussian means; performance drops more sharply when the memory buffer is very small. Future work may explore learning covariance matrices alongside means to increase expressive power.
Conclusion
DeepCCG demonstrates that alignment is the antidote to forgetting. By making the classifier instantaneously adaptable, the system prevents the encoder from receiving noisy signals, leading to more stable and effective continual learning.
Summary updated