Introduction
Task-Free Continual Learning (TFCL) represents a realistic yet challenging paradigm where models must learn from a continuous data stream without explicit task boundaries. While Dynamic Expansion Models (DEMs) effectively mitigate catastrophic forgetting by adding new components for novel data, they often suffer from "model bloat"—an unbounded growth in parameters that leads to significant computational inefficiency. The Lifelong Compression Mixture Model (LGMM) addresses this by introducing a novel mechanism to both grow and prune the model dynamically, ensuring a lean and efficient architecture.
System Architecture and Expansion
The LGMM framework is built upon three functional pillars designed to maintain a high-performance yet compact architecture.
To determine when the model requires additional capacity, LGMM employs a Maximum Mean Discrepancy (MMD) based expansion criterion. MMD provides a non-parametric way to compare distributions, allowing the model to detect shifts in data without relying on labels. If the statistical distance between the current memory buffer and the knowledge captured by existing components exceeds a threshold, the model triggers an expansion.
Complementing this is the Diversity-Aware Sample Selection (DivSS) strategy. Unlike standard recency-based buffers that use a "sliding window" approach, DivSS uses a Variational Autoencoder (VAE) to score the novelty of samples. It prioritizes storing samples that current experts find confusing, ensuring the memory buffer remains highly informative relative to what the model already knows.
The Knowledge Relationship Graph
The core innovation of LGMM is its data-free compression mechanism, which prevents indefinite growth by periodically evaluating the redundancy of its "experts."
The process involves building a Knowledge Relationship Graph. First, the model generates pseudo-samples from each component's internal generative model. It then calculates the Fréchet Inception Distance (FID) between pairs of components to quantify the similarity of their captured knowledge. These distances form the edges of a graph where nodes represent individual experts. A greedy pruning algorithm identifies and discards experts with high overlap, ensuring the remaining ensemble is diverse. This approach is "data-free" because it relies on internal generative capabilities rather than the original training data, making it highly scalable for long-term deployment.
Experimental Results
LGMM was benchmarked against several state-of-the-art models on standard datasets, including Split MNIST, CIFAR10/100, and Mini-ImageNet.
- Accuracy: On Split CIFAR100, LGMM (with DivSS) achieved a top accuracy of 26.03%, significantly outperforming other dynamic models such as CN-DPM (20.10%) and OCM (21.79%).
- Efficiency: In MNIST-SVHN cross-domain tasks, the discard mechanism successfully reduced the parameter count from 175M to 126M. This compression resulted in improved inference speeds with a negligible accuracy drop of approximately 1.5%.
- Robustness: The model proved resilient in "fuzzy boundary" scenarios, where data distributions shift gradually, demonstrating its readiness for unpredictable real-world data streams.
Conclusion
The LGMM demonstrates that for lifelong learning models, pruning is as vital as expansion. By leveraging a Knowledge Relationship Graph to identify and remove redundant components, the model maintains state-of-the-art accuracy while controlling architectural complexity. While the computational overhead of calculating FID scores remains a consideration for future optimization, the balance of growth and compression established here provides a robust foundation for efficient, task-free continual learning.
Summary updated