Lifelong Compression Mixture Model via Knowledge Relationship Graph

Fei Ye, Adrian G. Bors

Introduction

Task-Free Continual Learning (TFCL) represents a realistic yet challenging paradigm where models must learn from a continuous data stream without explicit task boundaries. While Dynamic Expansion Models (DEMs) effectively mitigate catastrophic forgetting by adding new components for novel data, they often suffer from "model bloat"—an unbounded growth in parameters that leads to significant computational inefficiency. The Lifelong Compression Mixture Model (LGMM) addresses this by introducing a novel mechanism to both grow and prune the model dynamically, ensuring a lean and efficient architecture.

System Architecture and Expansion

The LGMM framework is built upon three functional pillars designed to maintain a high-performance yet compact architecture.

Overview of the LGMM architecture, featuring MMD-based expansion and memory management.

To determine when the model requires additional capacity, LGMM employs a Maximum Mean Discrepancy (MMD) based expansion criterion. MMD provides a non-parametric way to compare distributions, allowing the model to detect shifts in data without relying on labels. If the statistical distance between the current memory buffer and the knowledge captured by existing components exceeds a threshold, the model triggers an expansion.

Complementing this is the Diversity-Aware Sample Selection (DivSS) strategy. Unlike standard recency-based buffers that use a "sliding window" approach, DivSS uses a Variational Autoencoder (VAE) to score the novelty of samples. It prioritizes storing samples that current experts find confusing, ensuring the memory buffer remains highly informative relative to what the model already knows.

The Knowledge Relationship Graph

The core innovation of LGMM is its data-free compression mechanism, which prevents indefinite growth by periodically evaluating the redundancy of its "experts."

The Knowledge Relationship Graph process for identifying and pruning redundant components.

The process involves building a Knowledge Relationship Graph. First, the model generates pseudo-samples from each component's internal generative model. It then calculates the Fréchet Inception Distance (FID) between pairs of components to quantify the similarity of their captured knowledge. These distances form the edges of a graph where nodes represent individual experts. A greedy pruning algorithm identifies and discards experts with high overlap, ensuring the remaining ensemble is diverse. This approach is "data-free" because it relies on internal generative capabilities rather than the original training data, making it highly scalable for long-term deployment.

Experimental Results

LGMM was benchmarked against several state-of-the-art models on standard datasets, including Split MNIST, CIFAR10/100, and Mini-ImageNet.

  • Accuracy: On Split CIFAR100, LGMM (with DivSS) achieved a top accuracy of 26.03%, significantly outperforming other dynamic models such as CN-DPM (20.10%) and OCM (21.79%).
  • Efficiency: In MNIST-SVHN cross-domain tasks, the discard mechanism successfully reduced the parameter count from 175M to 126M. This compression resulted in improved inference speeds with a negligible accuracy drop of approximately 1.5%.
  • Robustness: The model proved resilient in "fuzzy boundary" scenarios, where data distributions shift gradually, demonstrating its readiness for unpredictable real-world data streams.

Conclusion

The LGMM demonstrates that for lifelong learning models, pruning is as vital as expansion. By leveraging a Knowledge Relationship Graph to identify and remove redundant components, the model maintains state-of-the-art accuracy while controlling architectural complexity. While the computational overhead of calculating FID scores remains a consideration for future optimization, the balance of growth and compression established here provides a robust foundation for efficient, task-free continual learning.

Summary updated

Abstract

Task-Free Continual Learning (TFCL) represents a challenging scenario for lifelong learning because the model, under this paradigm, does not access any task information. The Dynamic Expansion Model (DEM) has shown promising results in this scenario due to its scalability and generalisation power. However, DEM focuses only on addressing forgetting and ignores minimizing the model size, which limits its deployment in practical systems. In this work, we aim to simultaneously address network forgetting and model size optimization by developing the Lifelong Compression Mixture Model (LGMM) equipped with the Maximum Mean Discrepancy (MMD) based expansion criterion for model expansion. A diversity-aware sample selection approach is proposed to selectively store a variety of samples to promote information diversity among the components of the LGMM, which allows more knowledge to be captured with an appropriate model size. In order to avoid having multiple components with similar knowledge in the LGMM, we propose a data-free component discarding mechanism that evaluates a knowledge relation graph matrix describing the relevance between each pair of components. A greedy selection procedure is proposed to identify and remove the redundant components from the LGMM. The proposed discarding mechanism can be performed during or after the training. Experiments on different datasets show that LGMM achieves the best performance for TFCL.
BibTeX
@inproceedings{ye2023lifelong,
  author = {Fei Ye and Adrian G. Bors},
  title = {{Lifelong Compression Mixture Model via Knowledge Relationship Graph}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i9.26292},
}
Figure