FedAD-Bench: A Unified Benchmark for Federated Unsupervised Anomaly Detection in Tabular Data

Ahmed Anwar, Brian Moser, Dayananda Herurkar, Federico Raue, Vinit Hegiste, Tatjana Legler, Andreas Dengel

TL;DR

FedAD-Bench is a new unified benchmarking framework designed to evaluate unsupervised deep anomaly detection algorithms in Federated Learning (FL) environments for tabular data. By introducing realistic data-splitting conventions and rigorous metric evaluations, the benchmark reveals that federated architectures can match—and sometimes even outperform—centralized models due to the inherent regularization effects of FL.

What Problem Does This Paper Address?

Anomaly detection is critical for identifying rare, high-stakes events like cyberattacks or medical anomalies. Because this data is often highly sensitive and distributed across different organizations, Federated Learning (FL) is a natural solution, allowing models to be trained collaboratively without sharing raw data.

However, evaluating anomaly detection within FL remains an open challenge. Traditional centralized benchmarks often include anomalies in the training set, which contradicts the goal of unsupervised anomaly detection (learning what is normal). Furthermore, standard evaluation metrics like AUROC can paint an overly optimistic picture in highly imbalanced datasets. FedAD-Bench addresses these gaps by establishing a standardized, reproducible framework tailored specifically for federated unsupervised anomaly detection on tabular data.

Methods

The authors introduce a comprehensive evaluation protocol built on three pillars:

  1. Redesigned Data Splitting: To properly evaluate unsupervised methods, the training set contains only normal data (inliers). The test set is composed of all the anomalies and the remaining 50% of the normal data. A stratified validation set with the same anomaly ratio is used to determine optimal decision thresholds.
  2. Federated Setup: The framework supports standard aggregation algorithms like FedAvg and FedProx (which adds a proximal term to limit local model deviation). It evaluates scalability across 3, 5, and 7 client setups.
  3. Algorithmic Diversity: The benchmark systematically evaluates five state-of-the-art deep learning models originally designed for centralized settings:
  4. DAE (Deep Auto Encoder)
  5. DSEBM (Deep Structured Energy Based Model)
  6. DeepSVDD (Deep Support Vector Data Description)
  7. NeuTraLAD (Neural Transformation Learning for Deep Anomaly Detection)
  8. MemAE (Memory Augmented Deep Auto Encoder)
Fig. 1. Unsupervised anomaly detection setup in both centralized and FL scenarios. Training is done on 50% of the normal data while the test set contains all anomalies as well as the other half of the normal data. Each client trains their local model on their own set of inliers. Afterward, the trained models are aggregated at the server using FedAvg and evaluated on the test set. The inference is based on the reconstruction error, where samples with high error values are considered anomalies.

Datasets

FedAD-Bench utilizes four popular tabular datasets spanning two critical domains:

  • Medical:
  • Arrhythmia: 452 samples, 274 features (anomaly ratio: 0.1460).
  • Thyroid: 3,772 samples, 6 features (anomaly ratio: 0.0246).
  • Cybersecurity:
  • KDDCUP10: A 10% subset containing 494,021 samples, 41 features (anomaly ratio: 0.1969).
  • NSL-KDD: An updated version of KDDCUP with 148,517 samples, 41 features (anomaly ratio: 0.4811).

Key Insights and Ideas

  • The FL Regularization Bonus: While centralized learning typically yields the best results, FL occasionally outperforms it. For instance, NeuTraLAD achieved significantly better results under FL on the cybersecurity datasets. The authors attribute this to the global aggregation step acting as a regularizer, which prevents local models from overfitting.
  • Simplicity Wins in Aggregation: Simpler architectures like the Deep Auto Encoder (DAE) maintain stable performance when aggregated. Conversely, highly complex models (like DSEBM) struggle, indicating a need for specialized FL aggregation strategies that account for complex model heuristics.
  • The AUROC Trap: The paper highlights how AUROC can be highly misleading in anomaly detection. Because normal samples vastly outnumber anomalies, the False Positive Rate denominator explodes, artificially inflating the AUROC score. The authors strongly advocate for using AUPR (Area Under the Precision-Recall Curve) as a more reliable metric.
  • Client Scalability Pressures: Increasing the number of clients (from 3 to 7) generally degrades performance because each client receives less data, causing local gradients to drift further from the global optimum before aggregation.

Key Results

  • Centralized vs. FL: In 8 out of 20 experiments, centralized learning won, but FL won in 4 cases. Notably, on the NSL-KDD dataset, NeuTraLAD's centralized AUROC was a poor 0.37, but shot up to 0.86 in the 3-client FL setup due to mitigated overfitting.
  • FedProx Benefits: Introducing the proximal term ($\mu$) in FedProx successfully controlled local overfitting on highly homogeneous datasets like KDD10, noticeably boosting performance for models like NeuTraLAD as $\mu$ increased from 0.01 to 1.0.

Limitations and Future Work

  • Loss of Local Heuristics: Standard FL weight aggregation algorithms strip away unique model heuristics between rounds. For example, MemAE relies on a memory module to store prototypes of normal patterns; averaging these weights at the server level dilutes their effectiveness. Specialized aggregation algorithms are needed to preserve these properties.
  • Dataset Homogeneity: Legacy benchmarks like KDD10 lack data diversity, leading to deceptively high performance across all models and offering fewer meaningful insights for modern applications. The authors call for the integration of more diverse, real-world tabular datasets in future iterations of the benchmark.

Summary updated

Abstract

The emergence of federated learning (FL) presents a promising approach to leverage decentralized data while preserving privacy. Furthermore, the combination of FL and anomaly detection is particularly compelling because it allows for detecting rare and critical anomalies (usually also rare in locally gathered data) in sensitive data from multiple sources, such as cybersecurity and healthcare. However, benchmarking the performance of anomaly detection methods in FL environments remains an underexplored area. This paper introduces FedAD-Bench, a unified benchmark for evaluating unsupervised anomaly detection algorithms within the context of FL. We systematically analyze and compare the performance of recent deep learning anomaly detection models under federated settings, which were typically assessed solely in centralized settings. FedAD-Bench encompasses diverse datasets and metrics to provide a holistic evaluation. Through extensive experiments, we identify key challenges such as model aggregation inefficiencies and metric unreliability. We present insights into FL's regularization effects, revealing scenarios in which it outperforms centralized approaches due to its inherent ability to mitigate overfitting. Our work aims to establish a standardized benchmark to guide future research and development in federated anomaly detection, promoting reproducibility and fair comparison across studies.
BibTeX
@misc{anwar2024fedad,
  author = {Ahmed Anwar and Brian Moser and Dayananda Herurkar and Federico Raue and Vinit Hegiste and Tatjana Legler and Andreas Dengel},
  title = {{FedAD-Bench: A Unified Benchmark for Federated Unsupervised Anomaly Detection in Tabular Data}},
  year = {2024},
  eprint = {2408.04442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.04442},
}
Figure