Abstract: This study explores the application of anomaly detection (AD) methods in imbalanced learning tasks, focusing on fraud detection using real online credit card payment data. We assess the performance of several recent AD methods and compare their effectiveness against standard supervised learning methods. Offering evidence of distribution shift within our dataset, we analyze its impact on the tested models’ performances. Our findings reveal that LightGBM exhibits significantly superior performance across all evaluated metrics but suffers more from distribution shifts than AD methods. Furthermore, our investigation reveals that LightGBM also captures the majority of frauds detected by AD methods. This observation challenges the potential benefits of ensemble methods to combine supervised, and AD approaches to enhance performance. In summary, this research provides practical insights into the utility of these techniques in real-world scenarios, showing LightGBM’s superiority in fraud detection while highlighting challenges related to distribution shifts.
Introduction
Fraud detection in online credit card payments is a high-stakes task characterized by two major hurdles: extreme class imbalance and significant distribution shifts. Fraudulent transactions often constitute less than 0.1% of total volume, making it difficult for standard algorithms to learn from such a sparse signal. Furthermore, fraudsters are adaptive; as security measures improve, they shift tactics, a phenomenon accelerated by global events like the COVID-19 pandemic. While Anomaly Detection (AD) methods are theoretically ideal for this environment—as they model "normality" rather than specific fraud patterns—their real-world efficacy compared to supervised learning remains a critical question.
Methodology
The study conducted a head-to-head evaluation using a massive proprietary dataset from a major French bank, comprising 192 million filtered transactions from 2018 through 2021. The researchers compared two main families of algorithms:
- Supervised Baseline: LightGBM, a Gradient Boosted Decision Tree (GBDT) framework known for its efficiency with tabular data.
- Anomaly Detection (AD): Models trained only on "normal" transactions, including shallow methods (Isolation Forest, ECOD, COPOD, KNN) and deep learning approaches (GOAD, NeuTraL-AD, NPT-AD).
To quantify the "drift" in fraud patterns over time, the authors utilized Optimal Transport Dataset Distance (OTDD) and visualization techniques (t-SNE and UMAP).
Quantifying Distribution Shift
The analysis focused on two specific regions: Country A (~3M payments) and Country B (~20M payments). Visualizations and OTDD metrics confirmed substantial shifts in data distribution between the 2018-2019 period and the 2020-2021 period, largely driven by changes in consumer behavior during the pandemic.
Key Findings
1. The Performance Gap
Across nearly all evaluated metrics (AUROC, AUPRC, and F1-score), LightGBM significantly outperformed AD methods. In Country A (2018), LightGBM achieved an AUPRC of 18.74%, while the best-performing AD method (NPT-AD) reached only 0.81%. This suggests that GBDTs are naturally resilient to imbalance because they focus on "hard-to-classify" samples during training.
2. The Robustness Paradox
AD methods like ECOD and COPOD demonstrated remarkable stability; their performance metrics barely shifted even when the underlying data distribution changed. However, this "robustness" came at the cost of much lower absolute performance. Even when LightGBM's performance degraded due to distribution shifts, it often still outperformed the stable AD models. A notable exception occurred in Country B, where the shift was most severe: Isolation Forest achieved a higher F1-score (1.3%) than LightGBM (0.48%), though LightGBM maintained its lead in AUROC.
3. Redundancy in Ensembles
A common strategy in fraud detection is to ensemble supervised and AD models to catch different types of fraud. However, the researchers found that 89.02% of the frauds caught by the best AD method were already captured by LightGBM, suggesting that such ensembles may offer limited marginal utility.
Conclusion
While Anomaly Detection offers theoretical resilience to novel fraud patterns, supervised GBDTs like LightGBM remain the superior choice for practical implementation due to their significantly higher detection rates. The primary challenge for future research lies in enhancing the robustness of supervised models to distribution shifts without sacrificing their high baseline performance.
Summary updated