Introduction
Continual Learning (CL) is defined by the Stability-Plasticity Dilemma: the need for a model to remain stable enough to retain old knowledge while being plastic enough to acquire new information. With the advent of large-scale Pre-Trained Models (PTMs), the dominant paradigm has shifted toward freezing the backbone and training lightweight modules like prompts or adapters. While this approach ensures stability, it often results in a "plasticity bottleneck," where the frozen features are insufficient for tasks significantly different from the pre-training data. Conversely, full fine-tuning often leads to "catastrophic forgetting" and destroys the general-purpose utility of the PTM.
The MIST Framework
This paper introduces Mutual Information-guided Sparse Tuning (MIST), a plug-and-play module designed to enhance plasticity without sacrificing stability. MIST operates as a lightweight pre-adaptation stage that selectively updates a tiny fraction of the PTM's parameters.
The core innovation lies in the use of a Mutual Information (MI) objective rather than standard Cross-Entropy (CE). The authors demonstrate that CE loss includes a term that inadvertently modifies the model's internal representation of the input distribution, leading to representation collapse. By using a supervised InfoNCE loss, MIST focuses solely on the relationship between inputs and labels.
Key Components of MIST:
- MI-Sensitive Parameter Identification: MIST calculates a Fisher Information Matrix based on the MI objective to identify parameters most critical for the new task.
- Sparse Selection: Only the top 5% of these sensitive parameters are targeted for updates.
- Gradient Dropout: During training, 90% of the gradients for these selected parameters are randomly zeroed out, resulting in an effective update of only ~0.5% of total parameters per step.
Experimental Evaluation
MIST was evaluated across diverse benchmarks, including CIFAR-100, ImageNet-R/A, and fine-grained datasets like CUB-200 and Cars-196. It was integrated into several state-of-the-art CL baselines, such as L2P, DualPrompt, and RanPAC.
Results and Generalization
The integration of MIST consistently yielded significant performance gains. For instance, on the fine-grained Cars-196 dataset, MIST improved the SimpleCIL baseline by 15.7%.
Beyond raw accuracy, MIST excels at preserving the PTM's general-purpose features. In zero-shot evaluations, where the model is tested on unrelated datasets after sequential training, MIST-tuned models retained significantly higher accuracy compared to those using full fine-tuning or other sparse methods. This demonstrates that MIST successfully navigates the stability-plasticity trade-off.
Conclusion
MIST addresses the plasticity bottleneck in PTM-based continual learning by enabling highly targeted, sparse updates guided by mutual information. By allowing a small portion of the backbone to adapt, MIST unlocks superior performance on challenging, out-of-distribution tasks while maintaining the stability required for long-term learning. Future research may explore more robust Fisher estimation techniques to handle intra-task distribution shifts or extend the framework to Large Language Models (LLMs).