DATE: Detecting Anomalies in Text via Self-Supervision of Transformers

Andrei Manolache, Florin Brad, Elena Burceanu

Abstract

Leveraging deep learning models for Anomaly Detection (AD) has seen widespread use in recent years due to superior performances over traditional methods. Recent deep methods for anomalies in images learn better features of normality in an end-to-end self-supervised setting. These methods train a model to discriminate between different transformations applied to visual data and then use the output to compute an anomaly score. We use this approach for AD in text, by introducing a novel pretext task on text sequences. We learn our DATE model end-to-end, enforcing two independent and complementary self-supervision signals, one at the token-level and one at the sequence-level. Under this new task formulation, we show strong quantitative and qualitative results on the 20Newsgroups and AG News datasets. In the semi-supervised setting, we outperform state-of-the-art results by +13.5% and +6.9%, respectively (AUROC). In the unsupervised configuration, DATE surpasses all other methods even when 10% of its training data is contaminated with outliers (compared with 0% for the others).

Introduction

Anomaly Detection (AD) is the challenge of identifying rare observations that deviate significantly from the norm. While deep learning has revolutionized AD in computer vision, text has remained a difficult domain due to its discrete and high-dimensional nature. Previous methods often relied on "shallow" techniques like One-Class SVMs or simple autoencoders that struggle to capture complex, contextual nuances. DATE (Detecting Anomalies in Text via ELECTRA) bridges this gap by leveraging self-supervised Transformers to learn a deep, robust representation of "normality" for text.

Methodology

DATE builds upon the ELECTRA architecture, which typically uses a Generator to replace tokens and a Discriminator to identify them. DATE introduces a dual-task framework to capture both local and global context:

  1. Replaced Token Detection (RTD): A token-level task where the model predicts whether each token in a sequence is "original" or "replaced."
  2. Replaced Mask Detection (RMD): A novel sequence-level task. Instead of random masking, the authors use $K$ predefined mask patterns. The discriminator must classify which of the $K$ patterns was used to corrupt the sentence.
The DATE training architecture, highlighting the dual RTD (token-level) and RMD (sequence-level) prediction heads.

By training on both signals, the model learns a holistic representation of text structure. For inference, DATE uses a "Pseudo Label" (PL) score based on the discriminator's confidence that tokens in an un-corrupted input are indeed original. This allows for efficient, single-pass anomaly scoring.

Results and Discussion

DATE was evaluated on the 20Newsgroups and AG News datasets, setting new benchmarks for text anomaly detection. In semi-supervised settings, DATE outperformed the previous state-of-the-art (CVDD) by +13.5% AUROC on 20Newsgroups and +6.9% on AG News.

One of the most significant findings is DATE's robustness to training data contamination. In unsupervised settings where the training data contains hidden anomalies, DATE maintains high performance.

Comparison of AUROC performance against outlier contamination in training data. DATE maintains superior performance even at 10% contamination.

Even with 10% contamination, DATE outperforms competing methods that were trained on perfectly clean data. The authors also noted that using a random generator for token replacement outperformed sophisticated learned generators, as the random noise provides a clearer boundary for the model to learn the distribution of normal data.

Conclusion

DATE demonstrates that joint token- and sequence-level self-supervision effectively captures the semantics of normality in text. Its efficiency and remarkable robustness to noisy training data make it a powerful framework for real-world applications. Future research may focus on optimizing mask pattern selection for specialized domains like medical or legal text.

Summary updated

Abstract

Leveraging deep learning models for Anomaly Detection (AD) has seen widespread use in recent years due to superior performances over traditional methods. Recent deep methods for anomalies in images learn better features of normality in an end-to-end self-supervised setting. These methods train a model to discriminate between different transformations applied to visual data and then use the output to compute an anomaly score. We use this approach for AD in text, by introducing a novel pretext task on text sequences. We learn our DATE model end-to-end, enforcing two independent and complementary self-supervision signals, one at the token-level and one at the sequence-level. Under this new task formulation, we show strong quantitative and qualitative results on the 20Newsgroups and AG News datasets. In the semi-supervised setting, we outperform state-of-the-art results by +13.5% and +6.9%, respectively (AUROC). In the unsupervised configuration, DATE surpasses all other methods even when 10% of its training data is contaminated with outliers (compared with 0% for the others).
BibTeX
@inproceedings{manolache2021date,
  author = {Andrei Manolache and Florin Brad and Elena Burceanu},
  title = {{DATE: Detecting Anomalies in Text via Self-Supervision of Transformers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=bHe1hnoh6o5},
}
Figure