TL;DR
This paper introduces FedTADBench, the first benchmark for federated time-series anomaly detection. It evaluates how existing anomaly detection algorithms perform when trained using federated learning, offering insights into the compatibility of different federated learning methods and the impact of data heterogeneity across clients.
What Problem Does This Paper Address?
Time-series anomaly detection is crucial for identifying unusual patterns in data, with applications ranging from manufacturing to healthcare. Traditionally, these models require large, centralized datasets for training. However, in many real-world scenarios, data is distributed across multiple devices or locations, and privacy concerns prevent centralizing it. Federated learning offers a solution by enabling model training on decentralized data without direct access to it. While federated learning is gaining traction, its effectiveness and compatibility with time-series anomaly detection algorithms remain largely unexplored. This paper aims to fill this gap by providing a comprehensive benchmark to evaluate these combinations.
Methods
The authors introduce FedTADBench, a benchmark designed to systematically evaluate time-series anomaly detection (TSAD) algorithms within a federated learning (FL) framework. The benchmark comprises:
- Five representative TSAD algorithms: DeepSVDD, LSTM-AE, USAD, GDN, and TranAD.
- Four popular FL methods: FedAvg, FedProx, SCAFFOLD, and MOON.
- Three popular TSAD datasets: SMD, SMAP, and PSM.
FedTADBench investigates three key aspects:
- TSAD performance under FL: How do standard TSAD algorithms fare when trained using FL compared to centralized training?
- FL method compatibility: Which FL methods are most suitable for TSAD tasks?
- Data heterogeneity impact: How does the distribution of data across clients affect the performance of federated TSAD?
The evaluation involves extensive experiments across various settings, including different data partitions on clients.
Datasets
The benchmark utilizes three publicly available time-series anomaly detection datasets:
- SMD (Server Machine Dataset): A 5-week-long dataset.
- SMAP (Soil Moisture Active Passive satellite): A dataset from NASA.
- PSM (Public Server Metrics): A dataset consisting of 26-dimensional time series data from application servers.
The paper also introduces the FedTADBench benchmark itself, which is a novel contribution that standardizes the evaluation of TSAD in federated settings.
Key Insights and Ideas
- Federated learning can be beneficial: In several instances, federated learning surprisingly outperformed centralized training. The authors hypothesize that FL might introduce a regularization effect that aids in training TSAD models.
- FL compatibility is not guaranteed: The performance of TSAD algorithms can vary significantly when combined with different FL methods. Some FL methods, like FedProx, which heavily regularizes local model updates, might hinder performance if the TSAD algorithm requires more adaptation to local data characteristics.
- Data heterogeneity has a limited impact: Experiments with different data partitions (using Dirichlet distributions) on clients showed that unbalanced data distributions did not significantly degrade performance. This suggests that the number of data points per client might be less critical than the overall training process in FL for TSAD.
- FedAvg and SCAFFOLD offer a balanced approach: These FL methods generally achieve good performance and are less prone to extreme performance drops compared to FedProx or MOON in certain scenarios.
Key Results
- Performance improvements with FL: For some TSAD algorithms (e.g., USAD on PSM), federated learning led to better AUC-ROC, AUC-PR, and F1 scores compared to isolated training.
- Varied FL method effectiveness: FedProx showed inconsistent performance, sometimes being the worst performer, particularly with algorithms like USAD that benefit from parameter adaptation. MOON, while offering relaxed constraints, often resulted in longer training times.
- Training time considerations: FedAvg was found to be the fastest among the federated learning methods evaluated, while MOON was the slowest due to its contrastive learning computations.
- Detailed metrics provided: The paper presents comprehensive results using AUC-ROC, AUC-PR, Precision, Recall, and F1-score (including adjusted metrics) across all combinations of TSAD algorithms, FL methods, and datasets.
Limitations and Future Work
- Limited number of FL methods and TSAD algorithms: While FedTADBench covers representative methods, future work could expand to include a wider range of algorithms.
- Focus on specific data characteristics: The benchmark primarily uses multivariate time series. Future research could explore its applicability to univariate time series or other data modalities.
- Guidance for model development: The authors suggest that their benchmark can guide the development of new federated learning frameworks specifically tailored for the characteristics of time-series anomaly detection.
The paper's open-source code is available, encouraging further research and reproducibility.
Summary updated