Out-of-Domain Intent Detection Considering Multi-Turn Dialogue Contexts

Hao Lang, Yinhe Zheng, Binyuan Hui, Fei Huang, Yongbin Li

TL;DR

Caro is a novel context-aware framework designed for Out-of-Domain (OOD) intent detection in multi-turn dialogue systems. By combining the information bottleneck principle to filter out conversational noise with a two-stage self-training strategy to mine unlabeled data, Caro achieves state-of-the-art performance, boosting the F1-OOD score by over 29% compared to previous best methods.

What Problem Does This Paper Address?

In practical dialogue systems (like virtual assistants or customer service bots), Intent Detection identifies what a user wants to achieve. However, deployed systems frequently encounter Out-of-Domain (OOD) intents—requests the system was never trained to handle. A robust system must correctly classify In-Domain (IND) tasks while gracefully rejecting OOD inputs.

While most existing OOD detection research focuses on single-turn utterances, real-world conversations are inherently multi-turn. Directly extending single-turn models to multi-turn settings introduces two major hurdles:

  1. The Long-Distance Obstacle: Long dialogue histories accumulate substantial noise (chitchat, irrelevant details) that can corrupt the learned intent representations.
  2. OOD Data Scarcity: Collecting and annotating multi-turn OOD examples for training is incredibly expensive and challenging.

The authors introduce Caro (Context-aware OOD intent detection) to tackle both issues simultaneously.

Methods

Caro's architecture relies on two core pillars to handle noisy contexts and data scarcity:

1. Multi-View Information Bottleneck (IB)

To strip away superfluous information from long dialogue histories, Caro leverages the information bottleneck principle. The system first concatenates the dialogue history and the current utterance, encoding them via a pre-trained BERT model. It then constructs two distinct "views" of this text sequence:

  • Global Pooling ($v_1$): A standard mean-pooling layer where every token embedding is weighted equally.
  • Adaptive Reception Field ($v_2$): An attention-like mechanism that dynamically weights token embeddings based on the input context.
Figure 1: Framework of Caro. For each input sample x = ⟨h, u⟩, two views v1(x) and v2(x) are obtained and a multi-view information bottleneck loss LIB is optimized to learn robust representations. A two-stage training process is introduced to mine OOD samples DO from unlabeled data DU, and optimize the cross entropy loss LCE with DO ∪DI

By optimizing a multi-view information bottleneck loss ($\mathcal{L}_{IB}$), Caro forces the model to maximize the mutual information shared between these two views (the core intent) while discarding the non-shared, superficial noise unique to either view. During training and inference, a dynamic gating mechanism aggregates these views into a unified representation.

2. Two-Stage Self-Training

To overcome the lack of labeled OOD data, Caro taps into readily available unlabeled data (a natural mixture of IND and OOD samples) using a bootstrapping approach:

  • Stage One: The model synthesizes pseudo-OOD samples by applying a "mixup" technique (convex combinations) to IND features, training a preliminary OOD detector.
  • Stage Two: This initial detector predicts labels on the unlabeled dataset. Samples confidently identified as OOD are mined to form a pseudo-OOD training set ($\mathcal{D}_O$), which is then used to jointly fine-tune the final $(k+1)$-way classifier.

Datasets

The framework was evaluated on two variants of the STAR dataset, a task-oriented dialogue dataset featuring 150 intents designed specifically to model long context dependencies:

  • STAR-Full: Contains all pre-processed samples from the original dataset. To simulate a realistic deployment, 30% of the IND samples and all OOD samples were stripped of their labels to form the unlabeled training partition ($\mathcal{D}_U$).
  • STAR-Small: A low-resource variant created by down-sampling the training set by 50% to test model robustness under data constraints.

Key Insights and Ideas

  • Information Compression is Key for Text: Traditional representation learning often tries to maximize all accessible information. Caro proves that compressing and explicitly discarding observational noise yields significantly more robust features for classification tasks plagued by long contexts.
  • Context Matters, But Only If Cleaned: The experiments show that models utilizing multi-turn history consistently beat single-turn variants. However, without an information bottleneck to filter the history, standard multi-turn models still choke on long-distance noise.
  • Unlabeled Data is an Effective Substitute for Human Annotation: By bootstrapping from synthesized pseudo-OOD data to mine real OOD data from an unlabeled pool, Caro bridges the performance gap toward an "Oracle" model trained on perfectly labeled data.

Key Results

Caro achieves massive performance leaps over competitive baselines across both datasets, evaluated using the macro F1-score over all classes (F1-All), In-Domain classes (F1-IND), and Out-of-Domain classes (F1-OOD):

  • STAR-Full: Caro achieved an F1-OOD score of 54.75%, outperforming the best baseline utilizing unlabeled data (ASS+MSP at 25.15%) by an absolute margin of 29.6%.
  • STAR-Small: In the low-resource setting, Caro maintained its dominance with an F1-OOD score of 46.78%, compared to the next best baseline at 21.27%.
  • Closer to the Upper Bound: Caro significantly closes the gap to the "Oracle" model (which has access to 100% perfectly labeled IND and OOD data), which scores 64.46% on STAR-Full.

A fine-grained analysis of the learned representations confirms that Caro successfully minimizes observational information while retaining high predictive power compared to alternative objectives like InfoMax or Multi-View Information (MVI).

Figure 2: Comparing representations obtained by different objectives on the STAR-Full dataset. A lower score means that the learned representa- tion discards more superficial information. See Appendix D for measurements used to produce the graph.

Limitations and Future Work

The primary limitation acknowledged by the authors is Caro's single-modality constraint. The framework is currently restricted entirely to textual inputs. In real-world deployments—such as smart home devices or robotics—dialogue systems must process multi-modal inputs including audio features, visual cues, and physical environment contexts. The authors note that expanding Caro to handle multi-modal, multi-turn contexts is a promising direction for future research.

Summary updated

Abstract

Out-of-Domain (OOD) intent detection is vital for practical dialogue systems, and it usually requires considering multi-turn dialogue contexts. However, most previous OOD intent detection approaches are limited to single dialogue turns. In this paper, we introduce a context-aware OOD intent detection (Caro) framework to model multi-turn contexts in OOD intent detection tasks. Specifically, we follow the information bottleneck principle to extract robust representations from multi-turn dialogue contexts. Two different views are constructed for each input sample and the superfluous information not related to intent detection is removed using a multi-view information bottleneck loss. Moreover, we also explore utilizing unlabeled data in Caro. A two-stage training process is introduced to mine OOD samples from these unlabeled data, and these OOD samples are used to train the resulting model with a bootstrapping approach. Comprehensive experiments demonstrate that Caro establishes state-of-the-art performances on multi-turn OOD detection tasks by improving the F1-OOD score of over $29\%$ compared to the previous best method.
BibTeX
@inproceedings{lang2024out,
  author = {Hao Lang and Yinhe Zheng and Binyuan Hui and Fei Huang and Yongbin Li},
  title = {{Out-of-Domain Intent Detection Considering Multi-Turn Dialogue Contexts}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=fUoojCBf0v},
}
Figure