DATE: Detecting Anomalies in Text via Self-Supervision of Transformers

Andrei Manolache, Florin Brad, Elena Burceanu

Introduction

Anomaly Detection (AD) in natural language processing is often described as a "needle in a haystack" problem. Identifying rare sequences that deviate from a complex semantic norm is challenging because "normality" is context-dependent and anomalies are frequently undefined until they occur. While computer vision has successfully utilized self-supervision (e.g., predicting image rotations) to learn features of normality, text lacks such intuitive geometric transformations. Previous text AD methods relied on shallow models or autoencoders that struggled to capture deep contextual meaning.

To bridge this gap, the authors of DATE (Detecting Anomalies in Text via ELECTRA) introduce a novel self-supervised framework that adapts Transformer-based pre-training specifically for the discrete domain of text anomaly detection.

The DATE Framework

The core of DATE is a "Generator-Discriminator" architecture inspired by ELECTRA, but modified with a dual-supervision objective designed to stabilize anomaly detection.

The DATE training architecture featuring the Generator-Discriminator setup and the dual RTD/RMD heads.

Architecture and Pretext Tasks

  1. Replaced Token Detection (RTD): This is a token-level task where a generator replaces certain words with alternatives. The discriminator must then identify which tokens are original and which were replaced.
  2. Replaced Mask Detection (RMD): This is a novel sequence-level task. The model is presented with a corrupted sentence and must identify which of $K$ predefined "mask patterns" (binary vectors) was used to create the corruption. This task forces the model to learn a global representation of the text, complementing the local focus of RTD.

Interestingly, the researchers found that a random generator—sampling words uniformly from the vocabulary—outperformed sophisticated learned generators for the specific goal of anomaly detection.

Anomaly Scoring via Pseudo Labels

To determine if a sequence is an anomaly during inference, DATE uses a simplified Pseudo Label (PL) score. Unlike previous methods that required multiple passes with different transformations, DATE feeds the original text through the discriminator once. The RTD head provides a probability for each token being "original." If the model is uncertain about the tokens in a clean sentence (resulting in a low aggregate PL score), the sequence is flagged as an anomaly.

Key Insights and Results

One of the most significant findings is the "Inlier Priority" phenomenon. Self-supervised models tend to learn the features of the majority data (inliers) much more effectively than outliers. During training, the model becomes an "expert" on what constitutes normality, causing anomalies to stand out naturally.

Evolution of PL score distributions for inliers and outliers from the start to the end of training across different datasets.

As shown in the distribution plots above, the model begins with overlapping distributions for inliers and outliers. By the end of training, the distributions separate significantly, with inliers consistently receiving higher PL scores. This separation is the foundation of the model's high performance.

Performance Highlights

  • State-of-the-Art Accuracy: On the 20Newsgroups dataset, DATE outperformed the previous best method (CVDD) by +13.5% AUROC. On the larger AG News dataset, it achieved a +6.9% improvement.
  • Efficiency: The model is highly viable for real-world applications, with an inference time of approximately 0.005 seconds per sample on a single GPU.
  • Robustness: Even when training data was "contaminated" with 10% outliers, DATE maintained superior performance compared to competing models trained on perfectly clean data.

Conclusion

DATE demonstrates that combining local token-level supervision with global sequence-level tasks can significantly enhance the ability of Transformers to detect semantic anomalies. By leveraging the Inlier Priority phenomenon and a simplified scoring mechanism, it provides a robust, efficient, and highly accurate solution for identifying outliers in complex text data.

Summary updated

Abstract

Leveraging deep learning models for Anomaly Detection (AD) has seen widespread use in recent years due to superior performances over traditional methods. Recent deep methods for anomalies in images learn better features of normality in an end-to-end self-supervised setting. These methods train a model to discriminate between different transformations applied to visual data and then use the output to compute an anomaly score. We use this approach for AD in text, by introducing a novel pretext task on text sequences. We learn our DATE model end-to-end, enforcing two independent and complementary self-supervision signals, one at the token-level and one at the sequence-level. Under this new task formulation, we show strong quantitative and qualitative results on the 20Newsgroups and AG News datasets. In the semi-supervised setting, we outperform state-of-the-art results by +13.5% and +6.9%, respectively (AUROC). In the unsupervised configuration, DATE surpasses all other methods even when 10% of its training data is contaminated with outliers (compared with 0% for the others).
BibTeX
@inproceedings{manolache2021date,
  author = {Andrei Manolache and Florin Brad and Elena Burceanu},
  title = {{DATE: Detecting Anomalies in Text via Self-Supervision of Transformers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=bHe1hnoh6o5},
}
Figure