Enhancing the generalization for Intent Classification and Out-of-Domain Detection in SLU

Yilin Shen, Yen-Chang Hsu, Avik Ray, Hongxia Jin

TL;DR

This paper introduces a novel "Domain-Regularized Module" (DRM) that enhances spoken language understanding (SLU) systems. By training only on in-domain data, DRM improves both intent classification accuracy and the crucial ability to detect out-of-domain (OOD) utterances, a common failure point in real-world voice assistants.

What Problem Does This Paper Address?

Spoken Language Understanding (SLU) systems, the brains behind voice assistants, are designed to understand user intents (e.g., "play music," "set a timer"). Traditionally, these systems are trained on a fixed set of "in-domain" (IND) utterances that cover expected commands. However, in real-world applications, users often utter commands that fall outside this predefined scope – these are "out-of-domain" (OOD) utterances. When an SLU system encounters an OOD utterance, it can lead to incorrect actions or frustrating failures, as illustrated by examples like a user asking to delete a playlist when the system only supports music playback.

Existing methods to address this often require collecting large amounts of OOD data, which is costly and difficult to make representative due to the infinite variety of potential OOD queries. This paper tackles the challenge of improving OOD detection without relying on external OOD data, using only the readily available IND training data. The goal is to build SLU systems that are not only accurate for known commands but also robust enough to gracefully reject unknown ones.

Methods

The core innovation is the Domain-Regularized Module (DRM). This module is designed to be a drop-in replacement for the final layer of existing neural network-based intent classifiers. The DRM works by decomposing the final "logits" (raw output scores before the final probability calculation) into two branches:

  • Classification Logits ($f_c$): These are responsible for predicting the IND intent classes, similar to a standard classifier.
  • Domain Logits ($f_d$): This is the novel component. It's trained to predict whether an utterance is IND or OOD. Since the training data is only IND, the DRM is trained with a "domain loss" that encourages $f_d$ to be close to 1 for all training samples. This effectively teaches the model to recognize the characteristics of IND data.

The DRM then combines these logits in a specific way (Equation 4 in the paper) that aims to reduce "overconfidence" in the classification predictions. By forcing the model to consider the "domain" of the utterance, it becomes less likely to assign high confidence to an OOD utterance that might superficially resemble an IND one.

Additionally, the paper proposes an extension called L-Mahalanobis, which leverages features from multiple layers of the neural network (specifically, transformer layers in BERT-like models) to compute an OOD detection score. This is an advancement over previous Mahalanobis distance methods that typically only used the final layer's features.

Datasets

The paper evaluates its approach on three benchmark SLU datasets and one in-house dataset:

  • CLINC: A widely used benchmark for OOD detection in SLU.
  • ATIS: A dataset focused on airline travel information.
  • Snips: A dataset covering music, book, and weather domains.
  • Movie (in-house): A custom dataset collected via crowdsourcing for movie-related queries.

For evaluation, the authors often treat datasets as mutually OOD to simulate diverse real-world scenarios.

Key Insights and Ideas

  • Leveraging IND data for OOD detection: The paper demonstrates that it's possible to significantly improve OOD detection by training only on IND data, challenging the common assumption that OOD data is essential.
  • Decomposing logits for robustness: Splitting the final output into classification and domain-specific components helps mitigate overconfidence and improves generalization.
  • Domain loss as a regularizer: The "domain loss" acts as a powerful regularizer, forcing the model to learn a more discriminative representation of IND data.
  • Multi-layer features for better OOD detection: Using features from various layers of deep models (like BERT) provides a richer representation for distinguishing IND from OOD utterances.

Key Results

  • The proposed DRM method, when combined with L-Mahalanobis, achieves state-of-the-art performance on the CLINC dataset for OOD detection, outperforming existing methods.
  • On various datasets, the DRM approach consistently improves intent classification accuracy by up to 0.8% compared to standard linear classifiers.
  • The DRM+L-Mahalanobis combination significantly reduces the overlap between IND and OOD score distributions, leading to better separation and detection.
  • Statistical significance tests confirm that the proposed methods are significantly better than baselines across multiple metrics.

Limitations and Future Work

  • Computational Cost: While DRM is a drop-in replacement, the L-Mahalanobis method involves processing features from multiple layers, which can increase computational cost.
  • Generalizability to other modalities: The paper focuses on SLU. While the authors suggest their work can contribute to other classification tasks (image, sound, vision-language), direct application and validation in those domains are future work.
  • Hyperparameter tuning: The paper mentions experimenting with a hyperparameter $\delta$ for clamping the domain logits, suggesting that careful tuning might be necessary for optimal performance.

The authors envision their work contributing to more robust classification systems across various domains, making AI assistants and other applications more reliable and user-friendly.

Summary updated

BibTeX
@inproceedings{shen2021enhancing,
  author = {Yilin Shen and Yen-Chang Hsu and Avik Ray and Hongxia Jin},
  title = {{Enhancing the generalization for Intent Classification and Out-of-Domain Detection in SLU}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=3HZihuwBWUS},
}
Figure