A Closer Look at Few-Shot Out-of-Distribution Intent Detection

Li-Ming Zhan, Haowen Liang, Lu Fan, Xiao-Ming Wu, Albert Y. S. LamCOLING 2022 · OpenReview

Abstract

We consider few-shot out-of-distribution (OOD) intent detection, a practical and important problem for the development of task-oriented dialogue systems. Despite its importance, this problem is seldom studied in the literature, let alone examined in a systematic way. In this work, we take a closer look at this problem and identify key issues for research. In our pilot study, we reveal the reason why existing OOD intent detection methods are not adequate in dealing with this problem. Based on the observation, we propose a promising approach to tackle this problem based on latent representation generation and self-supervision. Comprehensive experiments on three real-world intent detection benchmark datasets demonstrate the high effectiveness of our proposed approach and its great potential in improving state-of-the-art methods for few-shot OOD intent detection.

Introduction

In task-oriented dialogue systems, intent detection is the core mechanism for identifying user goals. However, real-world systems frequently encounter out-of-distribution (OOD) queries—inputs that do not fall into any pre-defined categories. While OOD detection is well-studied in high-resource settings, developers often face a "few-shot" scenario during early development, having only a handful of examples per intent.

A key challenge identified in this work is the "Conservative Boundary" trap. In few-shot settings, models tend to learn decision boundaries that are too tight around the sparse training points. This results in high precision but poor recall for in-distribution (ID) classes, as valid user queries that vary slightly from the training set are mistakenly rejected as OOD.

Comparison between real and learned decision boundaries in few-shot settings.

Methodology

To address the sparse data problem, we propose a two-stage framework that operates in the latent representation space of a pre-trained BERT model, avoiding the linguistic pitfalls of raw text augmentation.

Stage I: Latent ID Data Generation

We train a Denoising Autoencoder (DAE) on the available ID samples. The DAE's decoder is then used to generate synthetic ID examples by reconstructing noisy versions of existing representations. This process effectively "fills in" the semantic manifold around the sparse original data points, expanding the learned ID region.

Stage II: (K+1)-way Discriminative Training

To simulate OOD samples without external data, we utilize Random Convex Combinations (mixup-style interpolation). By interpolating between representations of different ID classes, we create "synthetic outliers." The model is then trained to classify known intents into $K$ categories and assign these synthetic interpolations to a $(K+1)$-th OOD class.

The two-stage framework involving latent denoising and (K+1)-way classification.

Experimental Evaluation

The framework was evaluated on three benchmarks: CLINC150 (150 intents), Banking (77 intents), and StackOverflow (20 intents). Few-shot settings were simulated by using only 5% or 10% of labeled data.

Key Results

  • Performance Gains: On CLINC150 (5% setting), our approach achieved a Macro F1-score of 62.19%, outperforming the ADB baseline by over 9%.
  • Robustness: The method showed consistent performance across different datasets and intent counts, unlike baselines that fluctuated significantly.
  • ID Improvement: Beyond OOD detection, the synthetic data generation also improved the model's ability to distinguish between known ID intents by providing a denser representation space.

Discussion and Conclusion

Our findings suggest that latent space augmentation is superior to text-based methods for few-shot intent detection. However, there is a "fidelity limit" to the DAE generator; performance degrades if too many synthetic examples (e.g., >100 per real sample) are generated, likely due to hallucinations that confuse the classifier. Future work will explore more advanced generative models like Variational Autoencoders (VAEs) or GANs to further refine the quality of synthetic ID representations.

Summary updated

Abstract

We consider few-shot out-of-distribution (OOD) intent detection, a practical and important problem for the development of task-oriented dialogue systems. Despite its importance, this problem is seldom studied in the literature, let alone examined in a systematic way. In this work, we take a closer look at this problem and identify key issues for research. In our pilot study, we reveal the reason why existing OOD intent detection methods are not adequate in dealing with this problem. Based on the observation, we propose a promising approach to tackle this problem based on latent representation generation and self-supervision. Comprehensive experiments on three real-world intent detection benchmark datasets demonstrate the high effectiveness of our proposed approach and its great potential in improving state-of-the-art methods for few-shot OOD intent detection.
BibTeX
@inproceedings{zhan2022closer,
  author = {Li-Ming Zhan and Haowen Liang and Lu Fan and Xiao-Ming Wu and Albert Y. S. Lam},
  title = {{A Closer Look at Few-Shot Out-of-Distribution Intent Detection}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=mEdW9LdNqJ},
}
Figure