Abstract
Real-life applications, heavily relying on machine learning, such as dialog systems, demand out-of-domain detection methods. Intent classification models should be equipped with a mechanism to distinguish seen intents from unseen ones so that the dialog agent is capable of rejecting the latter and avoiding undesired behavior. However, despite increasing attention paid to the task, the best practices for out-of-domain intent detection have not yet been fully established. This paper conducts a thorough comparison of out-of-domain intent detection methods. We prioritize the methods, not requiring access to out-of-domain data during training, gathering of which is extremely time- and labor-consuming due to lexical and stylistic variation of user utterances. We evaluate multiple contextual encoders and methods, proven to be efficient, on three standard datasets for intent classification, expanded with out-of-domain utterances. Our main findings show that fine-tuning Transformer-based encoders on in-domain data leads to superior results. Mahalanobis distance, together with utterance representations, derived from Transformer-based encoders, outperforms other methods by a wide margin and establishes new state-of-the-art results for all datasets. The broader analysis shows that the reason for success lies in the fact that the fine-tuned Transformer is capable of constructing homogeneous representations of in-domain utterances, revealing geometrical disparity to out of domain utterances. In turn, the Mahalanobis distance captures this disparity easily.
Introduction
In real-world dialog systems, models are trained to recognize a specific set of "in-domain" (ID) intents. However, users frequently provide "out-of-domain" (OOD) utterances that fall outside this set. If a system attempts to force these queries into existing categories, it results in erroneous behavior. Because gathering OOD data for training is labor-intensive and practically infinite, unsupervised methods are required. This research investigates how modern Transformer encoders, when paired with distance-based metrics, can effectively distinguish between seen and unseen intents.
Methodology
The study evaluates several text representation models and OOD scoring rules across three benchmarks: CLINC150, ROSTD, and SNIPS.
Encoders and Decision Rules
The researchers compared various encoders, including Bag-of-Words (BoW), LSTMs, and Transformers like BERT and RoBERTa. They tested three primary decision rules:
- Maximum Softmax Probability (MSP): A standard baseline rejecting inputs based on low confidence scores.
- Likelihood Ratio (LLR): Comparing inputs against a background noise model.
- Mahalanobis Distance: A metric that accounts for the distribution (covariance) of in-domain data, measuring the distance of a point from the class mean in terms of standard deviations.
The key innovation lies in applying the Mahalanobis distance to embeddings produced by a Transformer that has been fine-tuned on in-domain data.
Geometric Analysis: Why Mahalanobis Works
The success of this approach is rooted in how fine-tuning reshapes the embedding space. Fine-tuning forces in-domain utterances into dense, nearly spherical clusters that primarily occupy a low-dimensional subspace.
As illustrated in Figure 3, OOD utterances exhibit significantly higher variance in the "tail" components—dimensions that are typically noise for in-domain data. The Mahalanobis distance is exceptionally effective at capturing this geometric disparity. Additionally, fine-tuning ensures that class centroids are nearly orthogonal, preventing semantic "bleeding" between different intents.
Results and Performance
The combination of RoBERTa and Mahalanobis distance established new state-of-the-art results across all tested datasets.
Key performance metrics include:
- CLINC150: Achieved an AUROC of 98.4%, significantly higher than the BoW MSP baseline (91.5%) and previous benchmarks like BERT SNGP (96.9%).
- ROSTD: Achieved a near-perfect 0.5% False Positive Rate (FPR@95), indicating high precision in rejecting OOD queries.
- Efficiency: The method remains highly effective even with distilled models like DistilRoBERTa, making it suitable for production environments with latency constraints.
Conclusion
Fine-tuning Transformer-based encoders on in-domain data creates homogeneous representations that reveal clear geometrical disparities between ID and OOD utterances. Mahalanobis distance leverages these disparities to provide a robust, unsupervised OOD detection mechanism. While semantic overlap remains a challenge for future work, this approach significantly improves the reliability of intent classification in dialog systems.
The code is available at: https://github.com/huawei-noah/noah-research/tree/master/Maha_OOD
Summary updated