Estimating Soft Labels for Out-of-Domain Intent Detection

Hao Lang, Yinhe Zheng, Jian Sun, Fei Huang, Luo Si, Yongbin Li

Abstract

Detecting Out-of-Domain (OOD) intents is a critical challenge for dialogue systems, often hampered by the lack of real-world OOD training data. While synthetic "pseudo OOD" samples are commonly used to train detectors, treating them as "pure" outliers with hard labels can introduce noise and degrade performance. This paper introduces ASoul (Adaptive Soft Pseudo Labeling), a method that assigns soft labels to pseudo OOD samples based on their semantic similarity to In-Domain (IND) intents. By leveraging an embedding graph and a co-training framework, ASoul achieves state-of-the-art results across multiple benchmarks, particularly in data-scarce scenarios.

Introduction

Intent detection systems typically operate under a "closed-world" assumption, recognizing only a fixed set of In-Domain (IND) intents. However, real-world users frequently provide Out-of-Domain (OOD) queries. A robust system must identify these queries and reject them rather than misclassifying them into an IND category.

To train OOD detectors, researchers often synthesize "pseudo OOD" samples. Traditional methods assign these samples a "hard" one-hot label, treating them as 100% OOD. This approach is problematic because many synthetic samples are semantically close to IND intents. For example, a distorted query like "How much money do I earn" might be labeled as purely OOD despite its clear similarity to a "check salary" intent. This creates label noise that confuses the model and hurts its ability to generalize.

Figure 1: Comparison between traditional one-hot OOD labeling and ASoul's soft labeling approach.

Methods

The proposed ASoul framework treats pseudo OOD samples as unlabeled data that may contain IND semantic information. The method consists of three primary components:

  1. Embedding Graph Construction: Using supervised contrastive learning on IND data, the model develops a high-quality embedding space where semantically similar intents are clustered. A graph is then constructed where both IND and pseudo OOD samples serve as nodes, with edges representing semantic similarity.
  2. Graph-Smoothed Labeling: Following the "smoothness assumption"—that semantically similar samples should share similar labels—ASoul calculates a soft label for each pseudo OOD sample. This label is an aggregation of its neighbors' labels on the graph, allowing a sample to represent a mixture of IND intents and OOD characteristics.
  3. Co-Training Framework: To refine these labels, the system employs two classification heads ($g_1$ and $g_2$) that iteratively train each other. The prediction from one head is combined with the graph-smoothed label to serve as a target for the other, helping the model identify natural boundaries between intent classes.
Figure 2: The ASoul architecture, illustrating embedding graph construction, label smoothing, and the co-training process.

Results and Discussion

ASoul was evaluated on three widely used benchmarks: CLINC150, StackOverflow, and Banking. The method consistently outperformed baselines such as MSP, DOC, and ADB across various data-scarcity settings.

  • State-of-the-Art Performance: Combined with Feature Mixup (FM), ASoul set new performance records. On the CLINC150 dataset (25% IND setting), the overall F1-score improved from approximately 80% to 84.11%.
  • Robustness in Scarcity: The most significant gains occurred in scenarios with limited IND classes (25%), suggesting high utility for early-stage dialogue system development when data is most limited.
  • Semantic Insights: Analysis revealed that up to 39% of distorted pseudo OOD samples actually retained IND intent. ASoul's soft labeling (e.g., "15% Intent A, 20% Intent B, 65% OOD") prevents over-confidence and helps the model learn smoother decision boundaries.

Conclusion

ASoul demonstrates that treating pseudo OOD data as "soft" rather than "pure" outliers significantly enhances intent detection robustness. While currently focused on text and dependent on BERT-based encoders, the method is model-agnostic and shows promise for extension to multi-modal inputs and evolving intent sets in lifelong learning scenarios.

Summary updated

Abstract

Out-of-Domain (OOD) intent detection is important for practical dialog systems. To alleviate the issue of lacking OOD training samples, some works propose synthesizing pseudo OOD samples and directly assigning one-hot OOD labels to these pseudo samples. However, these one-hot labels introduce noises to the training process because some hard pseudo OOD samples may coincide with In-Domain (IND) intents. In this paper, we propose an adaptive soft pseudo labeling (ASoul) method that can estimate soft labels for pseudo OOD samples when training OOD detectors. Semantic connections between pseudo OOD samples and IND intents are captured using an embedding graph. A co-training framework is further introduced to produce resulting soft labels following the smoothness assumption, i.e., close samples are likely to have similar labels. Extensive experiments on three benchmark datasets show that ASoul consistently improves the OOD detection performance and outperforms various competitive baselines.
BibTeX
@inproceedings{lang2022estimating,
  author = {Hao Lang and Yinhe Zheng and Jian Sun and Fei Huang and Luo Si and Yongbin Li},
  title = {{Estimating Soft Labels for Out-of-Domain Intent Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=U0EJ_kMI0cp},
}
Figure