Ting Hua, Yilin Shen, Changsheng Zhao, Yen-Chang Hsu, Hongxia Jin. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.
Introduction
In the evolving landscape of Intelligent Personal Digital Assistants (IPDAs) such as Alexa or Google Assistant, new "domains"—categories of user requests like "Weather," "Music," or "Smart Home"—are added continuously. A primary challenge in this environment is enabling models to learn these new domains without suffering from catastrophic forgetting, where performance on previously learned tasks degrades significantly. While retraining on all accumulated data is a potential solution, it is computationally prohibitive at scale. Existing continual learning (CL) methods often rely on small subsets of old data or regularization terms, but these typically suffer from high hyperparameter sensitivity and performance instability, especially when new data distributions differ from the old.
Methodology: The CCFI Framework
To address these challenges, researchers at Samsung Research America developed CCFI (Continuous learning based on weight Consolidation and Fisher Information sampling). CCFI is designed to be a hyperparameter-free framework that stabilizes learning through two novel components:
- Fisher Information Sampling: Instead of traditional mean-vector sampling, CCFI utilizes Fisher Information to select "exemplars." This mathematical measure identifies data points most critical to the model's current parameter state, ensuring that the most vital internal knowledge is preserved even with a minimal memory footprint (as low as 1% of original data).
- Dynamical Weight Consolidation: Traditional methods like Elastic Weight Consolidation (EWC) require manual tuning of a fixed hyperparameter ($\lambda$) to balance old and new tasks. CCFI replaces this with a dynamical weighting mechanism that automatically calculates the optimal $\lambda$ at each training step by comparing current task loss against consolidation loss.
The system utilizes a frozen BERT backbone as a feature extractor, paired with an expandable linear classifier head that grows as new domains are introduced.
Experimental Evaluation
The researchers evaluated CCFI against state-of-the-art (SOTA) methods, including iCaRL and CoNDA, across three distinct datasets: a balanced 150-class intent classification benchmark, a highly imbalanced 73-domain real-world dataset, and a synthetic dataset for ablation studies.
Key Results and Discussion
CCFI demonstrated superior performance and stability across all benchmarks. Notably, on the 73-domain real-world dataset, CCFI maintained high accuracy while baselines like iCaRL saw performance drop to near zero as the number of classes increased.
Key insights from the evaluation include:
- Superior Accuracy: CCFI consistently outperformed SOTA methods, beating the next best method by up to 20% in average accuracy in certain scenarios.
- Robustness and Stability: Unlike baselines that exhibit erratic performance drops, CCFI shows a predictable, linear performance pattern, which is critical for industrial deployment.
- Efficiency: By freezing the BERT backbone and automating hyperparameter selection, CCFI significantly reduces the manual intervention and computational overhead typically associated with model fine-tuning.
Limitations and Future Work
While CCFI effectively prevents forgetting, the use of a frozen BERT backbone may limit the model's adaptability if future domains differ radically from the initial pre-training data. Future work will explore applying these dynamical weighting schemes to more complex NLU tasks, such as sequence labeling and text generation, and testing the limits of exemplar memory when scaling to thousands of domains.
Summary updated