TL;DR
FedAD-Bench is a new unified benchmarking framework designed to evaluate unsupervised deep anomaly detection algorithms in Federated Learning (FL) environments for tabular data. By introducing realistic data-splitting conventions and rigorous metric evaluations, the benchmark reveals that federated architectures can match—and sometimes even outperform—centralized models due to the inherent regularization effects of FL.
What Problem Does This Paper Address?
Anomaly detection is critical for identifying rare, high-stakes events like cyberattacks or medical anomalies. Because this data is often highly sensitive and distributed across different organizations, Federated Learning (FL) is a natural solution, allowing models to be trained collaboratively without sharing raw data.
However, evaluating anomaly detection within FL remains an open challenge. Traditional centralized benchmarks often include anomalies in the training set, which contradicts the goal of unsupervised anomaly detection (learning what is normal). Furthermore, standard evaluation metrics like AUROC can paint an overly optimistic picture in highly imbalanced datasets. FedAD-Bench addresses these gaps by establishing a standardized, reproducible framework tailored specifically for federated unsupervised anomaly detection on tabular data.
Methods
The authors introduce a comprehensive evaluation protocol built on three pillars:
- Redesigned Data Splitting: To properly evaluate unsupervised methods, the training set contains only normal data (inliers). The test set is composed of all the anomalies and the remaining 50% of the normal data. A stratified validation set with the same anomaly ratio is used to determine optimal decision thresholds.
- Federated Setup: The framework supports standard aggregation algorithms like FedAvg and FedProx (which adds a proximal term to limit local model deviation). It evaluates scalability across 3, 5, and 7 client setups.
- Algorithmic Diversity: The benchmark systematically evaluates five state-of-the-art deep learning models originally designed for centralized settings:
- DAE (Deep Auto Encoder)
- DSEBM (Deep Structured Energy Based Model)
- DeepSVDD (Deep Support Vector Data Description)
- NeuTraLAD (Neural Transformation Learning for Deep Anomaly Detection)
- MemAE (Memory Augmented Deep Auto Encoder)
Datasets
FedAD-Bench utilizes four popular tabular datasets spanning two critical domains:
- Medical:
- Arrhythmia: 452 samples, 274 features (anomaly ratio: 0.1460).
- Thyroid: 3,772 samples, 6 features (anomaly ratio: 0.0246).
- Cybersecurity:
- KDDCUP10: A 10% subset containing 494,021 samples, 41 features (anomaly ratio: 0.1969).
- NSL-KDD: An updated version of KDDCUP with 148,517 samples, 41 features (anomaly ratio: 0.4811).
Key Insights and Ideas
- The FL Regularization Bonus: While centralized learning typically yields the best results, FL occasionally outperforms it. For instance, NeuTraLAD achieved significantly better results under FL on the cybersecurity datasets. The authors attribute this to the global aggregation step acting as a regularizer, which prevents local models from overfitting.
- Simplicity Wins in Aggregation: Simpler architectures like the Deep Auto Encoder (DAE) maintain stable performance when aggregated. Conversely, highly complex models (like DSEBM) struggle, indicating a need for specialized FL aggregation strategies that account for complex model heuristics.
- The AUROC Trap: The paper highlights how AUROC can be highly misleading in anomaly detection. Because normal samples vastly outnumber anomalies, the False Positive Rate denominator explodes, artificially inflating the AUROC score. The authors strongly advocate for using AUPR (Area Under the Precision-Recall Curve) as a more reliable metric.
- Client Scalability Pressures: Increasing the number of clients (from 3 to 7) generally degrades performance because each client receives less data, causing local gradients to drift further from the global optimum before aggregation.
Key Results
- Centralized vs. FL: In 8 out of 20 experiments, centralized learning won, but FL won in 4 cases. Notably, on the NSL-KDD dataset, NeuTraLAD's centralized AUROC was a poor 0.37, but shot up to 0.86 in the 3-client FL setup due to mitigated overfitting.
- FedProx Benefits: Introducing the proximal term ($\mu$) in FedProx successfully controlled local overfitting on highly homogeneous datasets like KDD10, noticeably boosting performance for models like NeuTraLAD as $\mu$ increased from 0.01 to 1.0.
Limitations and Future Work
- Loss of Local Heuristics: Standard FL weight aggregation algorithms strip away unique model heuristics between rounds. For example, MemAE relies on a memory module to store prototypes of normal patterns; averaging these weights at the server level dilutes their effectiveness. Specialized aggregation algorithms are needed to preserve these properties.
- Dataset Homogeneity: Legacy benchmarks like KDD10 lack data diversity, leading to deceptively high performance across all models and offering fewer meaningful insights for modern applications. The authors call for the integration of more diverse, real-world tabular datasets in future iterations of the benchmark.
Summary updated