Abstract
Real-life applications, heavily relying on machine learning, such as dialog systems, demand for out-of-domain detection methods. Intent classification models should be equipped with a mechanism to distinguish seen intents from unseen ones so that the dialog agent is capable of rejecting the latter and avoiding undesired behavior. However, despite increasing attention paid to the task, the best practices for out-of-domain intent detection have not yet been fully established. This paper conducts a thorough comparison of out-of-domain intent detection methods. We prioritize the methods, not requiring access to out-of-domain data during training, gathering of which is extremely time- and labor-consuming due to lexical and stylistic variation of user utterances. We evaluate multiple contextual encoders and methods, proven to be efficient, on three common datasets for intent classification, expanded with out-of-domain utterances. Our main findings show that fine-tuning Transformer-based encoders on in-domain data leads to superior results. Mahalanobis distance, together with utterance representations, derived from Transformer-based encoders, outperform other methods by a wide margin(1-5% in terms of AUROC) and establish new state-of-the-art results for all datasets. The broader analysis shows that the reason for success lies in the fact that the fine-tuned Transformer is capable of constructing homogeneous representations of in-domain utterances, revealing geometrical disparity to out of domain utterances. In turn, the Mahalanobis distance captures this disparity easily.
Introduction
In specialized dialog systems, such as those used in banking, models are trained to recognize specific "in-domain" (ID) intents like checking balances or transferring funds. A significant challenge arises when users provide "out-of-domain" (OOD) inputs (e.g., asking for a recipe). Without robust OOD detection, classifiers often force these inputs into ID categories, leading to poor user experiences. Collecting representative OOD data for training is practically impossible, necessitating unsupervised methods that flag anomalies based solely on ID data characteristics.
Methods and Encoders
The study evaluates several text encoders and scoring rules. Encoders include traditional baselines (BoW, CNN, LSTM with GloVe) and modern Transformers (BERT, RoBERTa, and their distilled versions). The primary scoring rules compared are:
- Maximum Softmax Probability (MSP): Rejection based on low classifier confidence.
- Likelihood Ratio (LLR): Comparison against a background model.
- Mahalanobis Distance: A metric measuring how many standard deviations a point is from the mean of a distribution, accounting for feature correlations.
The core contribution is demonstrating how Mahalanobis distance effectively leverages the unique geometry of fine-tuned Transformer embeddings.
Key Insights: The Geometry of Intent
Fine-tuning Transformers on ID labels reshapes the embedding space into dense, homogeneous clusters where ID class centroids become mutually orthogonal. A critical discovery involves the variance of data components.
Using Principal Component Analysis (PCA), the authors observed that ID data is primarily explained by the first few dimensions. In contrast, OOD data exhibits significant variance in later, lower-variance components after fine-tuning. This "geometrical disparity" is precisely what the Mahalanobis distance captures, allowing for precise OOD detection.
Experimental Results
The combination of RoBERTa and Mahalanobis distance established a new state-of-the-art across multiple benchmarks, including CLINC150 and ROSTD.
On the CLINC150 dataset, this approach achieved an AUROC of 98.4%, representing a 1-5% improvement over previous methods like BERT SNGP. Notably, the Mahalanobis distance remains effective even with distilled models, making it suitable for resource-constrained deployments.
Conclusion
Fine-tuning Transformer encoders creates a structured embedding space that makes OOD utterances geometrically distinct. Mahalanobis distance is uniquely suited to identify these differences. While challenges remain regarding "near-OOD" samples that are semantically similar to ID intents, this approach provides a robust, unsupervised foundation for reliable dialog systems.
Summary updated