Abstract
Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines.
Introduction
In the field of Artificial Intelligence, "Continual Learning" (CL) is the ability for a model to learn a stream of tasks over time, mimicking human learning. However, most neural networks suffer from catastrophic forgetting—the tendency to abruptly overwrite weights from Task A when learning Task B.
Current solutions generally fall into two categories: replay-based methods, which store massive amounts of old data for retraining, and regularization-based methods, which penalize parameter changes. The latter often fails because it treats all information as equally important, making models either too rigid to learn or too flexible to remember. This paper introduces IDBR (Information Disentanglement Based Regularization) to address these limitations.
Methodology: The IDBR Framework
The authors propose a "divide and conquer" strategy for the model's hidden layers. Instead of a single representation vector, IDBR splits the hidden space into two distinct components: a Task-Generic Space (universal linguistic features) and a Task-Specific Space (features unique to a single domain).
To enforce this separation, the framework employs several key techniques:
- Auxiliary Tasks: Next Sentence Prediction (NSP) is used to train the generic space, as understanding sentence flow is a universal language skill. Task-ID Prediction is used to train the specific space, forcing it to capture what makes one task unique.
- Differential Regularization: When moving to a new task, the model applies a strict penalty to keep the generic space stable while allowing the specific space more room to adapt.
- Smart Replay: To boost performance, the model stores a tiny fraction (1%) of old data. Instead of random sampling, it uses K-Means clustering to select the most representative examples of each task.
Key Insights and Visualization
The core innovation of IDBR is the realization that not all features are equal. By isolating generic linguistic knowledge, the model can "freeze" the foundations of language while adapting to the nuances of a new task.
As shown in the t-SNE visualizations, the Task-Specific space exhibits clearer clustering by domain, whereas the Task-Generic space remains more integrated. This disentanglement allows the model to protect task-specific knowledge from being overwritten by subsequent tasks while maintaining a stable base of general language understanding.
Experimental Results
The model was evaluated on five large-scale benchmarks: AG News, Yelp, DBPedia, Amazon, and Yahoo! Answers. The researchers tested various task sequences to ensure robustness.
- Superior Performance: IDBR consistently outperformed standard baselines. In a 5-task sequence, IDBR achieved an average accuracy of 73.19%, compared to 69.67% for standard replay methods.
- Efficiency: IDBR reached performance levels comparable to heavy-duty models like LAMOL, but without requiring "Task IDs" during inference.
- Robustness: The "forgetting measure" remained remarkably low, proving the disentanglement effectively shielded old knowledge.
Conclusion
IDBR provides a scalable and effective solution for continual text classification. By disentangling representations and applying targeted regularization, it achieves state-of-the-art performance with minimal memory overhead. Future work may explore extending this strategy to more complex NLP tasks like sequence generation and automated tuning of regularization coefficients.
Summary updated