Abstract
Deployed real-world machine learning applications are often subject to uncontrolled and even potentially malicious inputs. Those out-of-domain inputs can lead to unpredictable outputs and sometimes catastrophic safety issues. Prior studies on out-of-domain detection require in-domain task labels and are limited to supervised classification scenarios. Our work tackles the problem of detecting out-of-domain samples with only unsupervised in-domain data. We utilize the latent representations of pre-trained transformers and propose a simple yet effective method to transform features across all layers to construct out-of-domain detectors efficiently. Two domain-specific fine-tuning approaches are further proposed to boost detection accuracy. Our empirical evaluations of related methods on two datasets validate that our method greatly improves out-of-domain detection ability in a more general scenario.
Introduction
Deep learning models are notoriously overconfident, often providing high-certainty predictions for inputs that are completely irrelevant to their training data—a phenomenon known as Out-of-Domain (OOD) input. While OOD detection is critical for safety and reliability, most existing methods rely on supervised data, requiring labeled in-domain examples to learn the boundaries of "normal" behavior. In many real-world scenarios, such as topic modeling or summarization, these labels are unavailable. This paper addresses the challenge of unsupervised OOD detection: identifying whether a test sample belongs to the same distribution as an unlabeled in-domain dataset.
Methodology
The proposed approach leverages the rich representations inside pre-trained Transformers through a three-pronged strategy that combines feature extraction with domain-specific fine-tuning.
Mahalanobis Distance Features (MDF)
OOD signals can manifest at various levels of abstraction, from syntactic anomalies in middle layers to vocabulary shifts in bottom layers. To capture these signals without the computational burden of high-dimensional feature concatenation, the authors propose calculating the Mahalanobis distance for each layer independently. This process transforms high-dimensional embeddings into a compact vector (e.g., 12 dimensions for a 12-layer model), which is subsequently processed by a One-Class SVM (OC-SVM) to produce an anomaly score.
Domain-Specific Fine-Tuning
To further enhance detection accuracy, two fine-tuning methods are introduced:
- In-domain Masked Language Modeling (IMLM): The Transformer is fine-tuned on unlabeled in-domain data using the standard masked language modeling task. This encourages the model's features to concentrate around the specific characteristics of the in-domain distribution.
- Binary Classification with Auxiliary Dataset (BCAD): A small subset of general pre-training data (e.g., Wikipedia) is used as "pseudo-OOD" samples. A binary classifier is trained to distinguish between specific in-domain data and this general auxiliary data, sharpening the model's sensitivity to outliers.
Results and Evaluation
The method was evaluated on two distinct OOD challenges: cross-corpus detection (using SST as in-domain) and cross-intent detection (using CLINIC150).
The combination of IMLM, BCAD, and MDF achieved state-of-the-art results. On the SST dataset, the model reached an AUROC of 99.9% using RoBERTa. On the more challenging CLINIC150 dataset, it achieved an AUROC of 84.4%, a significant improvement over the 64.6% baseline from a standard single-layer BERT model.
A key insight from the experiments is that the "best" layer for detection varies by dataset—top layers for SST and middle layers for CLINIC150—validating the multi-layer MDF approach. Furthermore, MDF proved more effective than simple pooling strategies, which often diluted the OOD signal by treating all layers with equal importance.
Conclusion
This work demonstrates that unsupervised OOD detection can be highly effective by utilizing the multi-layer representations of pre-trained Transformers. By reducing complex latent states to Mahalanobis distances and employing self-supervised fine-tuning, the proposed method provides a robust and efficient solution for identifying out-of-domain inputs in label-scarce environments. Future work may explore these techniques in active learning and data augmentation pipelines.
Summary updated