Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding

Ting Hua, Yilin Shen, Changsheng Zhao, Yen-Chang Hsu, Hongxia Jin

Abstract

Domain classification is the fundamental task in natural language understanding (NLU), which often requires fast accommodation to new emerging domains. This constraint makes it impossible to retrain all previous domains, even if they are accessible to the new model. Most existing continual learning approaches suffer from low accuracy and performance fluctuation, especially when the distributions of old and new data are significantly different. In fact, the key real-world problem is not the absence of old data, but the inefficiency to retrain the model with the whole old dataset. Is it potential to utilize some old data to yield high accuracy and maintain stable performance, while at the same time, without introducing extra hyperparameters? In this paper, we proposed a hyperparameter-free continual learning model for text data that can stably produce high performance under various environments. Specifically, we utilize Fisher information to select exemplars that can "record" key information of the original model. Also, a novel scheme called dynamical weight consolidation is proposed to enable hyperparameter-free learning during the retrain process. Extensive experiments demonstrate that baselines suffer from fluctuated performance and therefore useless in practice. On the contrary, our proposed model CCFI significantly and consistently outperforms the best state-of-the-art method by up to 20% in average accuracy, and each component of CCFI contributes effectively to overall performance.

Introduction

In the fast-paced world of Virtual Assistants (like Alexa or Siri), new "domains"—new sets of user intents and topics—are added constantly. Ideally, these systems should learn new skills without forgetting previous ones. However, deep learning models famously suffer from catastrophic forgetting: when trained on new data, they often overwrite the knowledge needed for previous tasks.

Most current continual learning methods fall into two traps. First is the Tuning Trap, where they rely on complex hyperparameters (like the "importance" weight of old tasks) that require expert manual tuning for every new dataset. Second is the Stability Trap, where performance fluctuates wildly when new data differs significantly from old data. Researchers from Samsung Research America have proposed CCFI (Continuous learning based on weight Consolidation and Fisher Information sampling) to address these issues without manual hyperparameter tuning.

Methodology

The CCFI framework introduces two core innovations to create a robust, tuning-free model:

Fisher Information Sampling

Instead of picking "average" data points to save in memory, CCFI uses the Fisher Information Matrix. This mathematical tool identifies which specific data points are most critical for maintaining the current values of the model's parameters. These "high-information" samples act as a more effective "record" of what the model learned previously.

Algorithm 1: Construction of exemplar set using Fisher Information.

Dynamical Weight Consolidation

Traditional methods like Elastic Weight Consolidation (EWC) use a fixed penalty to prevent the model from changing important weights. CCFI replaces this with a dynamic penalty. It automatically calculates the ratio between the loss on new data and the "consolidation loss" (the penalty for changing old weights), allowing the model to adjust its own learning balance on the fly. The architecture utilizes a frozen BERT backbone as a feature extractor and an expandable linear layer as the classifier.

Experimental Setup

The researchers evaluated CCFI on three distinct benchmarks:

  1. CLINC150: A balanced dataset featuring 150 different intents.
  2. 73-domain dataset: A real-world, highly imbalanced dataset representing actual product environments.
  3. Synthetic Dataset: A controlled environment with 10 completely separable classes used for ablation studies to prove the effectiveness of individual components.

Results and Discussion

CCFI consistently outperformed state-of-the-art baselines like iCaRL and CoNDA. In class-incremental learning tests, it maintained an average accuracy lead of up to 20%.

Performance comparison of CCFI against baselines across overall, new, and old task accuracies.

Key insights from the results include:

  • Predictable Decay: Unlike fine-tuning methods that show sudden drops, CCFI exhibits a linear, predictable performance pattern, which is crucial for engineering maintenance and estimating when a full retrain is necessary.
  • Efficiency: CCFI achieves high accuracy even with access to as little as 1% of the original data, proving that the quality of exemplars (selected via Fisher Information) is more important than quantity.
  • Zero-Tuning Success: The dynamical approach converged quickly and smoothly, whereas fixed-weight strategies often failed to converge or fluctuated wildly.

Conclusion

CCFI represents a significant step toward robust, "set-and-forget" continuous learning for NLU. By automating hyperparameter selection and improving exemplar quality through Fisher Information, it provides a stable and high-performing solution for evolving virtual assistant domains. Future work may explore unfreezing the backbone for greater adaptability in fundamentally different tasks.

Reference: Hua, T., Shen, Y., Zhao, C., Hsu, Y. C., & Jin, H. (2022). Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding. arXiv:2201.01420v1.

Summary updated

Abstract

Domain classification is the fundamental task in natural language understanding (NLU), which often requires fast accommodation to new emerging domains. This constraint makes it impossible to retrain all previous domains, even if they are accessible to the new model. Most existing continual learning approaches suffer from low accuracy and performance fluctuation, especially when the distributions of old and new data are significantly different. In fact, the key real-world problem is not the absence of old data, but the inefficiency to retrain the model with the whole old dataset. Is it potential to utilize some old data to yield high accuracy and maintain stable performance, while at the same time, without introducing extra hyperparameters? In this paper, we proposed a hyperparameter-free continual learning model for text data that can stably produce high performance under various environments. Specifically, we utilize Fisher information to select exemplars that can "record" key information of the original model. Also, a novel scheme called dynamical weight consolidation is proposed to enable hyperparameter-free learning during the retrain process. Extensive experiments demonstrate that baselines suffer from fluctuated performance and therefore useless in practice. On the contrary, our proposed model CCFI significantly and consistently outperforms the best state-of-the-art method by up to 20% in average accuracy, and each component of CCFI contributes effectively to overall performance.
BibTeX
@inproceedings{hua2021hyperparameter,
  author = {Ting Hua and Yilin Shen and Changsheng Zhao and Yen-Chang Hsu and Hongxia Jin},
  title = {{Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=tSc1fCPq6Si},
}
Figure