Abstract
Task-Oriented Dialogue Systems (TDS) typically struggle with catastrophic forgetting when learning new domains sequentially. This paper introduces TCDR (Contrastive Distillation on Internal Representations), a framework designed for the challenging Class Incremental Learning (CIL) setting where task labels are unknown at test time. By combining contrastive learning on intermediate model layers with a dynamic embedding update strategy, TCDR maintains performance on old tasks while acquiring new ones. Experimental results across seven domains demonstrate that TCDR significantly outperforms existing baselines in both language fluency and information retrieval accuracy.
Introduction
Most AI dialogue systems are trained in a static manner, learning all domains simultaneously. However, real-world systems must often learn new skills or domains (e.g., adding "Hotel Booking" to a system that already knows "Weather") over time. The primary hurdle is catastrophic forgetting, where learning a new task overwrites the weights used for previous tasks. While previous research addressed Task Incremental Learning (where the system is told which domain it is handling), this paper tackles the more realistic Class Incremental Learning (CIL) scenario. In CIL, the system must handle any task it has ever learned without being explicitly told which one the user is referencing.
Methodology
The authors build their approach on the Global-to-local Memory Pointer (GLMP) architecture and introduce two key innovations:
- Contrastive Distillation on Internal Representations (TCDR): Instead of matching only the final output of the old model, TCDR focuses on the model's "internal thoughts." It applies a Contrastive Loss ($L_{CL}$) to the global memory representation ($h^G$) and the local memory decoder's state ($h^L$). This ensures the model's underlying reasoning logic remains stable even as it learns new domains.
- Dynamic Update Mechanism: To prevent representation drift, the authors use a momentum strategy to update embeddings. When a new task arrives, they primarily update parameters related to that task while keeping previous parameters fixed or updating them very slowly. This explicitly shields old knowledge from being overwritten.
Experimental Setup
The researchers evaluated TCDR on seven distinct tasks derived from three popular benchmark datasets: Stanford Multi-domain Dialogue (SMD), Multi-WOZ 2.1, and CamRest. These cover domains such as navigation, weather, scheduling, and hospitality, providing a rigorous test for cross-domain learning.
Results and Discussion
TCDR outperformed all major baselines across the board. After learning all seven tasks, performance was measured using BLEU (language fluency) and Entity F1 (information accuracy). TCDR achieved an average of 8.97 BLEU and 39.31 Entity F1, whereas the base GLMP model without distillation dropped to 4.65 BLEU and 26.49 Entity F1, confirming massive forgetting in the absence of the proposed framework.
Key insights from the results include:
- Internal Consistency: Matching internal representations ($h^G$ and $h^L$) is more effective for complex dialogue than standard logit-based distillation, as it preserves the logic required to query external databases.
- Selective Plasticity: The dynamic update mechanism effectively maintains a "memory" of previous vocabularies and entities by freezing or slowly updating relevant embedding spaces.
Conclusion
TCDR demonstrates that preserving internal consistency is vital for maintaining performance in incremental dialogue learning. While the method shows significant promise, future work may explore how this approach scales to hundreds of tasks and whether the reliance on a memory buffer can be reduced or eliminated.
Summary updated