TL;DR
This paper introduces a novel, lightweight Non-negative Matrix Factorization (NMF) approach for textual anomaly detection. By replacing the standard squared error with a Correntropy-induced metric and optimizing at the row level, their method achieves performance comparable to state-of-the-art deep learning models, even with limited data and computational resources.
What Problem Does This Paper Address?
Detecting anomalies in text, such as hate speech or offensive language, is challenging due to imbalanced datasets where anomalies are rare and non-uniform. Traditional machine learning methods and even recent transformer-based deep learning models, while effective, often require extensive pre-training, large datasets, and significant computational power, making them inflexible for new domains or resource-constrained environments. This paper aims to bridge this gap by proposing a more efficient and adaptable anomaly detection method.
Methods
The core of the proposed approach is a Non-negative Matrix Factorization (NMF) model. The authors build upon existing NMF techniques for textual anomaly detection, which typically frame the problem as reconstructing a document-term matrix and identifying outliers as residual noise.
Their key innovations are:
- Correntropy-Induced Metric (CIM): Instead of the standard squared error in the NMF objective function, they use a CIM based on a Gaussian kernel. This metric is argued to be more robust to outliers in textual data.
- Row-Level Optimization: Following Du et al. (2012), they optimize the CIM at the row level of the residual matrix, rather than at the individual entry level. This allows for a more holistic assessment of topical anomalies within segments of text.
- Lightweight Architecture: The approach avoids the need for pre-training or attention layers, making it computationally less demanding and more adaptable to new domains.
They compare their proposed method, termed R-NMF, against a baseline NMF model (H-NMF) and a state-of-the-art transformer model (DATE).
Datasets
The authors evaluated their models on four publicly available datasets:
- 20Newsgroups: A collection of approximately 20,000 newsgroup documents across 20 topical subgroups.
- Reuters-21578: A dataset of news stories from 1987, indexed with categories.
- WikiPeople: A subset of Wikipedia containing articles categorized as "living people."
- Hate Speech Dataset: A dataset of forum posts from Stormfront, a white-supremacy forum, where the "hate" class constitutes about 11% of the corpus.
For each dataset, they created experimental settings with three different outlier concentrations: 0.01, 0.025, and 0.05. They specifically aimed to select inlier and outlier topics with reasonably close topical content to create challenging classification scenarios.
Key Insights and Ideas
- Correntropy's Suitability for Text: The paper suggests that the Correntropy-induced metric is a better fit for textual anomalies than traditional squared error, likely due to its robustness to noisy or unusual data points common in text.
- Row-Level Anomaly Detection: Optimizing anomaly detection at the row level (document segment) rather than individual word counts provides a more nuanced understanding of topical intrusions.
- Lightweight Models Can Compete: The R-NMF model demonstrates that effective anomaly detection doesn't always require massive deep learning models, offering a viable alternative for resource-constrained scenarios.
- Adaptability is Key: The lack of pre-training makes R-NMF highly adaptable to new domains and datasets, a significant advantage over transformer models.
Key Results
- Competitive Performance: The R-NMF model achieved performance comparable to, and in many cases superior to, the state-of-the-art DATE transformer model across various datasets and outlier concentrations.
- Outperforming Baselines: R-NMF consistently outperformed the H-NMF baseline, particularly on the Reuters and WikiPeople datasets.
- Hate Speech Efficacy: For the hate speech dataset, both NMF-based models outperformed the DATE model, with R-NMF showing particular strength at lower outlier concentrations.
- Computational Efficiency: The R-NMF model required significantly fewer computational resources (3 CPU cores) compared to the DATE model (2 GPUs and 6 cores), making it a more practical choice.
- Robustness to Small Concentrations: R-NMF performed well even with very small outlier concentrations, a crucial aspect for real-world applications where anomalies are often rare.
Limitations and Future Work
- Limitations: The authors acknowledge that anomaly detection models can have imperfect Precision and Recall, meaning they might misclassify texts. They highlight the need for human review in high-risk contexts.
- Future Work: The authors plan to conduct further experiments on new anomaly detection datasets, including those containing hate speech and offensive language. They also suggest exploring the impact of different hyper-parameter settings and potentially incorporating more advanced regularization techniques.
Summary updated