TL;DR
This paper investigates how pre-training data impacts textual anomaly detection, specifically in a "k-classes-out" setting where one class is normal and others are anomalies. They demonstrate that simple models like OC-SVM, when trained with word representations learned only from the anomaly detection task's training data, can outperform complex, state-of-the-art deep learning models.
What Problem Does This Paper Address?
Anomaly Detection (AD) in text is challenging because anomalies are inherently hard to define and characterize. Most existing AD methods in Natural Language Processing (NLP) are framed as "out-of-domain" detection within established classification tasks. However, the NLP field heavily relies on pre-trained language models and embeddings. The impact of this pre-training data on AD tasks, especially those lacking explicit supervision, has been largely overlooked. This paper aims to fill this gap by systematically studying how the source of word representations (pre-trained on general corpora vs. trained specifically on the AD task data) affects AD performance, particularly in a "k-classes-out" scenario.
Methods
The authors focus on the "k-classes-out" setting, where one class from a multi-class dataset is designated as normal, and all other classes are treated as anomalies. They experiment with two main AD models:
- OC-SVM (One-Class Support Vector Machine): A traditional, non-neural baseline model.
- CVDD (Contextualized Word Embeddings for Anomaly Detection): A model that uses attention mechanisms to learn topic representations from word embeddings.
They explore various word representations:
- Static Embeddings:
- FastText (FTLarge): Pre-trained on large external corpora (Wikipedia, Common Crawl).
- FTD and FTC: FastText embeddings trained specifically on the AD task's training data (FTD for the full training set, FTC for only the normal class).
- PPMI: Pointwise Mutual Information, a traditional method, reduced via SVD. PPMID and PPMIC are versions trained on the full training data and normal class data, respectively.
- Contextual Embeddings:
- ELECTRA: Off-the-shelf pre-trained model.
- DATE: A model based on ELECTRA that learns contextual representations and is trained for anomaly detection. They also experiment with ELECTRA representations trained locally on each class.
The core novelty lies in comparing models using representations trained on general data versus those trained locally on the AD task's training data.
Datasets
The paper uses three publicly available textual datasets:
- 20 Newsgroups: A collection of news articles from 20 different topics.
- AG News: A topic classification dataset with 4 classes, derived from news articles.
- RNCP (Répertoire National des Certifications Professionelles): A dataset of French certification titles, noted for its short document lengths.
These datasets are adapted to the "k-classes-out" setting by designating one class as normal and the rest as anomalies.
Key Insights and Ideas
- Local Learning Trumps General Pre-training: The most significant finding is that word representations trained solely on the AD task's training data (e.g., FTD, FTC, PPMIC) consistently lead to better anomaly detection performance, even for simple models like OC-SVM, compared to representations pre-trained on massive, general-purpose corpora (e.g., FTLarge).
- Simplicity Can Be Superior: A basic OC-SVM model, when paired with well-chosen, locally trained representations, can outperform sophisticated deep learning models like CVDD and DATE. This suggests that the complexity of deep models might not always be necessary or even beneficial in this specific AD setting.
- Dataset Size Matters for Representation Choice: For OC-SVM, the performance of class-based representations (like FTC) becomes more competitive with dataset-based representations (like FTD) as the dataset size increases. This implies that with more data, learning class-specific nuances becomes more feasible.
- Linear Kernels Shine: For OC-SVM, a linear kernel often yields the best results, suggesting that the geometry of the locally learned representations is well-suited for linear separation, and more complex kernels might lead to overfitting.
- Contextual Embeddings Not Always Better: While contextual embeddings from DATE showed promise, combining them with OC-SVM did not consistently outperform other methods, and in some cases, led to worse results. This highlights that the specific training objective of AD models matters.
Key Results
- OC-SVM with locally trained representations achieved state-of-the-art results on all three datasets, outperforming CVDD and DATE. For instance, on 20 Newsgroups, OC-SVM with FTD representations achieved an AUC of 89.8±0.01, significantly better than DATE's 59.2±0.1.
- FTD representations (FastText trained on the full AD training data) generally outperformed FTLarge (pre-trained on external data) for both OC-SVM and CVDD.
- PPMIC often beat PPMID on 20 Newsgroups and AG News, suggesting that statistics derived solely from class data can be more representative for simpler kernels.
- RNCP dataset proved challenging: AUC scores were generally lower across all models, likely due to the short document lengths.
Limitations and Future Work
- Limited Datasets: The authors acknowledge that only a handful of AD-specific textual datasets exist. They used existing classification datasets adapted to the AD setting.
- Focus on Semantic Shifts: The "k-classes-out" setting primarily addresses semantic shifts, not background shifts.
- Contextual Representations: While they experimented with contextual representations (DATE, ELECTRA), they found that combining them with OC-SVM did not consistently yield competitive results, and some contextual representations did not perform well.
- Future Work: The authors intend to extend their work to more challenging textual AD tasks and explore larger, more recent language models for representation learning. They also suggest that model adaptation methods for semantic anomaly detection warrant further investigation.
Summary updated