Retrieval Augmented Deep Anomaly Detection for Tabular Data

Hugo Thimonier, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan

Abstract

Deep learning for tabular data has garnered increasing attention in recent years, yet employing deep models for structured data remains challenging. While these models excel with unstructured data, their efficacy with structured data has been limited. Recent research has introduced retrieval-augmented models to address this gap, demonstrating promising results in supervised tasks such as classification and regression. In this work, we investigate using retrieval-augmented models for anomaly detection on tabular data. We propose a reconstruction-based approach in which a transformer model learns to reconstruct masked features of normal samples. We test the effectiveness of KNN-based and attention-based modules to select relevant samples to help in the reconstruction process of the target sample. Our experiments on a benchmark of 31 tabular datasets reveal that augmenting this reconstruction-based anomaly detection (AD) method with sample-sample dependencies via retrieval modules significantly boosts performance. The present work supports the idea that retrieval modules are useful to augment any deep AD method to enhance anomaly detection on tabular data.

Introduction: The Challenge of Tabular Data

While deep learning has revolutionized image and text processing, tabular data remains a stubborn frontier. Standard deep models often struggle with the heterogeneous nature of tables—where one column might be a category and the next a continuous measurement—and often fail to outperform classical tree-based models like Random Forests.

In the realm of Anomaly Detection (AD), the challenge is even steeper. Most algorithms focus on one of two things:

  1. Feature-feature dependencies: How do different columns relate to each other? (e.g., "If Age is 5, then Income should be 0.")
  2. Sample-sample dependencies: How does this row compare to its neighbors? (e.g., "This data point looks nothing like the others in this cluster.")

The authors argue that to catch all types of anomalies—global outliers, local deviations, and broken dependencies—a model needs to do both. Current deep AD methods often neglect the sample-sample relationship, leaving a performance gap that this paper aims to bridge.

Methodology: Retrieval-Augmented Reconstruction

The proposed framework, referred to as NPT-AD, utilizes a Masked Reconstruction task. The model is given a row of data with some features hidden (masked) and must predict those values. Since the model is trained exclusively on "normal" data, it learns the underlying patterns of standard samples. Consequently, it struggles to reconstruct "anomalous" data, resulting in a high error score that serves as an anomaly signal.

The proposed architecture for Retrieval-Augmented Deep Anomaly Detection, featuring a Transformer Encoder and an External Retrieval Module.

As shown in Figure 1, the architecture consists of two main components:

  • The Transformer Encoder: This handles the internal logic of the row, learning how features interact with one another (feature-feature dependencies).
  • The External Retrieval Module: Inspired by Retrieval-Augmented Generation (RAG), this module queries the training set for every sample to find the $k$ most relevant "helper" samples.

Several retrieval strategies were evaluated, with attention-bsim emerging as the most effective. Unlike standard dot-product attention, attention-bsim utilizes a similarity score based on the negative Euclidean distance between samples in the latent space. This retrieved information is blended with the target sample’s representation before the final reconstruction step.

Experimental Results

The researchers evaluated NPT-AD on an extensive benchmark of 31 tabular datasets, including 28 from the ODDS (Outlier Detection DataSets) collection and real-world datasets like Fraud, Campaign, and Backdoor.

Average F1 scores and rankings across 31 datasets, showing the superior performance of the proposed NPT-AD method.

As illustrated in Figure 4, the results were conclusive:

  • Performance Boost: Adding the attention-bsim retrieval module improved the average F1-score by 4.3% and the AUROC by 1.2% compared to a vanilla Transformer.
  • Robustness: NPT-AD achieved the highest average F1 score (68.8) and the best average rank (3.1) across all 31 datasets, outperforming established baselines like GOAD, NeuTraL-AD, and classical methods like Isolation Forest.
  • Synthetic Validation: In a custom 3D synthetic test, the augmented model was the only one capable of identifying both "Type 1" (local) and "Type 2" (dependency) anomalies with high accuracy, whereas the Transformer and KNN-based models each failed on one type.

Key Insights and Discussion

  • The "Why" of Retrieval: Different anomalies require different "lenses." A vanilla Transformer is adept at catching "dependency anomalies" (broken feature relationships) but fails at "local anomalies" (points in unusual locations relative to neighbors). Retrieval provides the missing local context.
  • Non-Parametric Power: By using an external retrieval module, the model becomes "non-parametric." It doesn't just rely on internal weights; it effectively uses the entire training dataset as a reference library during inference.
  • Distance vs. Dot-Product: Interestingly, standard dot-product attention performed worse than distance-based similarity (attention-bsim). In tabular spaces, the physical distance between neighbors appears more informative than simple projections.

Limitations and Future Work

Despite its success, the model has a higher memory and computational footprint during inference because it must retrieve neighbors from the training set. For massive datasets, this could become a bottleneck. Additionally, performance is sensitive to hyperparameters like $k$ (number of helpers) and $\lambda$ (trust weight for retrieved info). Future research may explore "plugging in" this retrieval mechanism to other deep AD methods beyond reconstruction-based architectures.

Summary updated

BibTeX
@inproceedings{thimonier2024retrieval,
  author = {Hugo Thimonier and Fabrice Popineau and Arpad Rimmel and Bich-Liên Doan},
  title = {{Retrieval Augmented Deep Anomaly Detection for Tabular Data}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2024},
  url = {https://openreview.net/forum?id=K4a1tbyYfY},
}
Figure