Abstract
In virtual assistant (VA) systems, it is critical to reject or redirect user queries that fall outside the system's supported capabilities. One of the most accurate approaches for out-of-scope (OOS) rejection involves combining it with intent classification using similarity-based embeddings from transformer-based sentence encoders. Typically, these encoders are fine-tuned using cross-entropy loss. However, recent research indicates that while cross-entropy is effective for classification, it tends to disperse in-scope embeddings across the entire embedding space, causing potential overlaps with OOS data. To mitigate this, we propose regularizing the cross-entropy loss with an in-scope embedding reconstruction loss learned via an auxiliary autoencoder. Our method achieves a 1-4% improvement in the area under the precision-recall curve (AUPR) for OOS rejection without compromising intent classification performance.
Introduction
Virtual Assistants (VAs) operate within a defined "scope" of intents, such as "set an alarm" or "check the weather." A primary challenge in Natural Language Understanding (NLU) is Out-of-Scope (OOS) detection: the system's ability to identify when a user query falls outside these predefined categories.
Standard fine-tuning practices utilize cross-entropy loss ($L_{CE}$), which excels at creating separation between classes. However, this often results in embeddings being scattered throughout the mathematical space. When the model encounters an OOS query, that query may inadvertently land near these dispersed in-scope clusters, leading to false positives. This study introduces a method to keep the "neighborhood" of in-scope embeddings tight, making the boundary between the known and the unknown much clearer.
Methodology
Joint Training Objective
The proposed approach utilizes a joint training objective that balances classification accuracy with embedding compactness. The model architecture includes a standard transformer backbone with two heads:
- Classification Head: Uses standard cross-entropy loss ($L_{CE}$) to distinguish between in-scope intents.
- Autoencoder Head: An auxiliary reconstruction loss ($L_{AE}$) that compresses the sentence embedding into a 16-dimensional bottleneck before attempting to reconstruct it.
The final loss function is defined as: $L = (1 - \alpha)L_{CE} + \alpha L_{AE}$
By minimizing reconstruction error, the transformer is forced to project in-scope samples into a structured, compact neighborhood. This regularization ensures that the model does not just learn the boundaries between classes, but also the boundaries of the entire known world.
Inference
During inference, the autoencoder and classification heads are discarded. The system calculates the Mahalanobis distance—a measure of how many standard deviations a point is from the distribution of an intent cluster. If the distance to the nearest cluster exceeds a specific threshold, the query is rejected as OOS.
Experimental Setup
The researchers evaluated the method across four datasets representing different scales and domains:
- CLINC150: A standard NLU benchmark with 150 intent classes.
- StackOverflow: 20 categories derived from technical forum posts.
- MTOP: A task-oriented dialogue dataset covering 11 domains.
- Car Assistant: A large-scale internal dataset from Cerence Inc., featuring 600,000 training utterances and 200,000 OOS test samples from real-world automotive usage.
Results and Discussion
Performance Gains
The proposed method demonstrated a consistent 1% to 4% improvement in the Area Under the Precision-Recall curve (AUPR) for OOS detection across all datasets. Crucially, this improvement did not degrade intent classification accuracy; for instance, accuracy on CLINC150 remained stable at 95.8%.
Embedding Dispersion
To validate the theory, the authors measured "embedding dispersion" using the trace of the global covariance matrix. The results confirmed that the autoencoder successfully reduced dispersion, creating a tighter manifold for in-scope data. This effect was most pronounced in the large-scale Car Assistant dataset, suggesting the regularizer scales effectively with data volume.
Comparison with Large Language Models
In a side-by-side comparison with GPT-3.5, the fine-tuned transformer model actually outperformed the larger model in OOS detection for these specific tasks. This suggests that for specialized edge-case detection, a smaller, specifically regularized model can be more effective than a general-purpose LLM.
Conclusion
By incorporating an auxiliary reconstruction loss, we can regularize sentence transformers to maintain compact in-scope embeddings. This "tight neighborhood" approach significantly enhances a virtual assistant's ability to detect out-of-scope queries without sacrificing classification performance. Future work will explore the application of this method to compositional queries and multi-intent detection.
Summary updated