Centroid Distance Distillation for Effective Rehearsal in Continual Learning

Daofeng Liu, Fan Lyu, Linyan Li, Zhenping Xia, Fuyuan Hu

TL;DR

This paper tackles the problem of "catastrophic forgetting" in continual learning by proposing a novel "Centroid Distance Distillation" (CDD) method. CDD uses centroids to intelligently sample data for rehearsal and distills the relationships between these centroids, rather than raw data features, to prevent models from forgetting past knowledge while learning new tasks.

What Problem Does This Paper Address?

Continual Learning (CL) aims to enable machine learning models to learn sequentially from a stream of tasks without forgetting previously acquired knowledge. A major challenge in CL is "catastrophic forgetting," where learning a new task overwrites knowledge from older tasks. A common solution is "rehearsal," which involves storing a small subset of data from previous tasks and retraining on it alongside new data. However, this rehearsal data can be biased, leading to "continual domain drift" – a shift in the model's internal feature representations for old tasks. This drift can cause forgetting and hinder the learning of new tasks. This paper specifically addresses how to mitigate this continual domain drift during rehearsal.

Methods

The proposed method, Centroid Distance Distillation (CDD), has two main components:

  • Centroid-based Sampling: Instead of random sampling or using all available data, CDD first constructs "centroids" which represent the centers of data clusters for each class. A "centroid caching mechanism" is employed to store representative samples near these centroids. This mechanism helps to reduce bias in the sampled rehearsal data by selecting samples that are more representative of the original data distribution for each class.
  • Centroid Distance Distillation: This is the core innovation. Instead of storing and distilling the actual features of the rehearsal samples (which can be memory-intensive and still prone to drift), CDD stores and distills the pairwise distances (specifically, cosine similarity) between the centroids of different classes. This "centroid distance matrix" captures the relative relationships between classes. During subsequent task learning, a "Centroid Distillation loss" is used to ensure the new model's centroid distances match the stored ones, effectively guiding the model to maintain the learned relationships and prevent domain drift.

Datasets

The proposed method was evaluated on four standard continual learning datasets:

  • Permuted MNIST: A variant of MNIST where input pixels are randomly permuted for each task.
  • Split CIFAR: A version of CIFAR100 where classes are split into disjoint tasks.
  • Split CUB: A dataset derived from the CUB-200-2011 dataset, with classes split into disjoint tasks.
  • Split AWA: An incremental version of the Animals with Attributes dataset, where categories are repeatable across tasks.

Key Insights and Ideas

  • Domain Drift as the Root Cause: The paper frames catastrophic forgetting not just as a loss of accuracy, but as a consequence of "continual domain drift" in the feature space. This reframing leads to a more targeted solution.
  • Centroids as Representative Anchors: Using class centroids as anchors for sampling rehearsal data is a clever way to ensure that the stored data is more representative of the original class distributions, thus reducing bias.
  • Distilling Relationships, Not Raw Features: The most significant insight is distilling the relationships between centroids (their distances) rather than the features themselves. This is more memory-efficient and directly targets the structural changes in the feature space that cause domain drift.
  • Trade-off in Centroid Distance: The paper highlights that the choice of the centroid distance threshold (€) is important, with a larger distance leading to less class overlap but potentially more drift, and a smaller distance leading to more overlap but better representation.

Key Results

  • Improved Accuracy: The proposed CDD method consistently outperforms existing state-of-the-art methods across all four datasets in terms of Average Accuracy (Ат). For example, on Split CIFAR, the proposed method achieved 69.65% AT, outperforming methods like A-GEM (61.28%) and ER (63.97%).
  • Reduced Forgetting: The method also shows significant improvements in Forgetting Measure (FT) and Long-Term Remembering (LTR) metrics, indicating better retention of old knowledge.
  • Mitigated Domain Drift: Qualitative results using t-SNE visualizations demonstrate that the feature distributions of old tasks remain much more stable with CDD compared to baseline methods, even after learning multiple new tasks.
  • Efficiency: By storing only centroid distances, the method is memory-efficient compared to storing raw features.

Limitations and Future Work

  • Hyperparameter Sensitivity: The paper acknowledges that the performance is sensitive to hyperparameter choices, particularly the centroid distance threshold (€) and cache size (γ). While they perform grid searches, finding optimal values might require careful tuning for different datasets.
  • Computational Overhead: While memory is reduced, the initial computation of centroids and their distances adds some computational overhead, especially in the early stages of learning.
  • Future Work: The authors suggest that combining their method with Feature Distillation (FD) loss further improves performance, hinting at potential avenues for exploring hybrid approaches. They also implicitly suggest further research into more robust centroid estimation and sampling strategies.

Summary updated

Abstract

Rehearsal, retraining on a stored small data subset of old tasks, has been proven effective in solving catastrophic forgetting in continual learning. However, due to the sampled data may have a large bias towards the original dataset, retraining them is susceptible to driving continual domain drift of old tasks in feature space, resulting in forgetting. In this paper, we focus on tackling the continual domain drift problem with centroid distance distillation. First, we propose a centroid caching mechanism for sampling data points based on constructed centroids to reduce the sample bias in rehearsal. Then, we present a centroid distance distillation that only stores the centroid distance to reduce the continual domain drift. The experiments on four continual learning datasets show the superiority of the proposed method, and the continual domain drift can be reduced. Our code is available at https://github.com/Daofeng-liu/CDD-R.
BibTeX
@inproceedings{liu2023centroid,
  author = {Daofeng Liu and Fan Lyu and Linyan Li and Zhenping Xia and Fuyuan Hu},
  title = {{Centroid Distance Distillation for Effective Rehearsal in Continual Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=hji6kEosMi},
}

Loading related content...

Figure