Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLU

Yilin Shen, Yen-Chang Hsu, Avik Ray, Hongxia Jin

Abstract

Intent classification is a major task in spoken language understanding (SLU). Since most models are built with pre-collected in-domain (IND) training utterances, their ability to detect unsupported out-of-domain (OOD) utterances has a critical effect in practical use. Recent works have shown that using extra data and labels can improve the OOD detection performance, yet it could be costly to collect such data. This paper proposes to train a model with only IND data while supporting both IND intent classification and OOD detection. Our method designs a novel domain-regularized module (DRM) to reduce the overconfident phenomenon of a vanilla classifier, achieving a better generalization in both cases. Besides, DRM can be used as a drop-in replacement for the last layer in any neural network-based intent classifier, providing a low-cost strategy for a significant improvement. The evaluation on four datasets shows that our method built on BERT and RoBERTa models achieves state-of-the-art performance against existing approaches and the strong baselines we created for the comparisons.

Introduction

Spoken Language Understanding (SLU) systems, such as those powering modern voice assistants, typically operate under a "closed-world assumption." They are trained on a fixed set of in-domain (IND) utterances to map user queries to predefined intents. However, real-world usage frequently involves out-of-domain (OOD) queries—tasks the system was never designed to handle. Standard neural networks often exhibit overconfidence, incorrectly forcing these OOD queries into IND categories rather than rejecting them.

While one could train a binary classifier to reject OOD inputs, collecting a representative OOD dataset is practically impossible given the infinite nature of the "open world." This research addresses the challenge of training a model to accurately classify IND intents and reliably detect OOD queries using only IND training data.

Proposed Methodology

The paper introduces two primary technical contributions designed to enhance transformer-based models like BERT and RoBERTa.

Domain-Regularized Module (DRM)

The DRM serves as a drop-in replacement for the standard linear classification layer. It bifurcates the final network logits into two distinct branches:

  • Domain Logits ($f_d$): Captures the probability that an utterance belongs to the in-domain distribution.
  • Classification Logits ($f_c$): Represents standard class probabilities.

The final output is derived as $f_c / f_d$. During training, a "domain loss" is applied alongside the standard cross-entropy loss. Since all training data is IND, this loss encourages $f_d$ to approach 1. This mathematical structure penalizes overconfidence on data that deviates from the IND distribution.

The Domain-Regularized Module (DRM) architecture splits logits into domain and classification branches to regularize confidence.

L-Mahalanobis Distance

For inference-time OOD detection, the authors propose L-Mahalanobis, an adaptation of the Mahalanobis distance metric. To address the semantic divergence between different transformer layers, features from each layer are passed through a non-linear tanh function. This aligns them into a shared semantic space before they are aggregated to compute the final distance score.

Experimental Evaluation

The proposed methods were evaluated across four diverse datasets: CLINC (a dedicated OOD benchmark), ATIS (airline travel), Snips (multi-domain), and an in-house Movie QA dataset.

Results and Analysis

The combination of DRM and L-Mahalanobis achieved state-of-the-art performance. Notably, DRM not only improved OOD detection but also enhanced IND classification accuracy. This suggests that the domain loss encourages the model to learn more precise, tightly clustered representations of IND data.

Comprehensive evaluation of OOD detection and intent classification across BERT and RoBERTa models.

On the CLINC dataset using RoBERTa, the DRM + L-Mahalanobis approach achieved an Equal Error Rate (EER) of 6.11% and a False Positive Rate (FPR95) of 7.63%, significantly outperforming standard Mahalanobis baselines (8.40% EER and 11.82% FPR95).

Conclusion

The Domain-Regularized Module provides a computationally efficient and highly effective method for improving the robustness of SLU systems. By disentangling domain membership from class probability, the model achieves superior generalization without the need for costly OOD data collection. Future work may explore applying DRM to other modalities such as computer vision and sound classification.

Summary updated

Abstract

Intent classification is a major task in spoken language understanding (SLU). Since most models are built with pre-collected in-domain (IND) training utterances, their ability to detect unsupported out-of-domain (OOD) utterances has a critical effect in practical use. Recent works have shown that using extra data and labels can improve the OOD detection performance, yet it could be costly to collect such data. This paper proposes to train a model with only IND data while supporting both IND intent classification and OOD detection. Our method designs a novel domain-regularized module (DRM) to reduce the overconfident phenomenon of a vanilla classifier, achieving a better generalization in both cases. Besides, DRM can be used as a drop-in replacement for the last layer in any neural network-based intent classifier, providing a low-cost strategy for a significant improvement. The evaluation on four datasets shows that our method built on BERT and RoBERTa models achieves state-of-the-art performance against existing approaches and the strong baselines we created for the comparisons.
BibTeX
@inproceedings{shen2021enhancing,
  author = {Yilin Shen and Yen-Chang Hsu and Avik Ray and Hongxia Jin},
  title = {{Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLU}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=3HZihuwBWUS},
}
Figure