Benchmarking IoT Time-Series AD with Event-Level Augmentations

Dmitry Zhevnenko, Ilya Makarov, Aleksandr Kovalenko, Fedor Meshchaninov, Anton Kozhukhov, Vladislav Travnikov, Makar Ippolitov, Kirill Yashunin, Iurii Katser

Abstract

Anomaly detection (AD) for safety-critical IoT time series should be judged at the event level: reliability and earliness under realistic perturbations. Yet many studies still emphasize point-level results on curated base datasets, limiting value for model selection in practice. We introduce an evaluation protocol with unified event-level augmentations that simulate real-world issues: calibrated sensor dropout, linear and log drift, additive noise, and window shifts. We also perform sensor-level probing via mask-as-missing zeroing with per-channel influence estimation to support root-cause analysis. We evaluate 14 representative models on five public anomaly datasets (SWaT, WADI, SMD, SKAB, TEP) and two industrial datasets (steam turbine, nuclear turbogenerator) using unified splits and event aggregation. There is no universal winner: graph-structured models transfer best under dropout and long events; density/flow models work well on clean stationary plants but can be fragile to monotone drift; spectral CNNs lead when periodicity is strong; reconstruction autoencoders become competitive after basic sensor vetting; predictive/hybrid dynamics help when faults break temporal dependencies but remain window-sensitive. The protocol also informs design choices: on SWaT under log drift, replacing normalizing flows with Gaussian density reduces high-stress F1 from ~0.75 to ~0.57, and fixing a learned DAG gives a small clean-set gain (~0.5-1.0 points) but increases drift sensitivity by ~8x.

Introduction

Anomaly detection (AD) in safety-critical cyber-physical systems (CPS)—such as nuclear power plants or water treatment facilities—is a high-stakes task. However, current evaluation methodologies for machine learning models are often disconnected from operational reality. Most benchmarks rely on clean, curated data and point-level metrics, which fail to capture the importance of detecting contiguous anomaly events or the impact of real-world sensor degradation. In practice, sensors drift, fail, or become noisy, and models must remain robust without constant recalibration. This paper introduces a deployment-first evaluation protocol that tests models against realistic stressors to provide a more accurate assessment of their utility in the field.

Methods and Evaluation Protocol

The proposed protocol shifts the focus from point-level accuracy to event-level reliability through three primary components:

  1. Event-Level Aggregation: Utilizing unified data splits to evaluate a model's ability to detect contiguous anomalous episodes rather than isolated data points.
  2. Offline-Calibrated Stress Suite: Models are subjected to simulated perturbations—including Gaussian noise, linear/log drift, and sensor dropouts—calibrated to the nominal variance of the dataset. Crucially, models are frozen before testing to simulate zero test-time calibration.
  3. Sensor-Level Probing: A "mask-as-missing" technique identifies the influence of individual channels, helping engineers detect "toxic" sensors that may degrade overall performance.
Figure 3: Visualization of training and test cases. The training data represents the healthy state, while the test case illustrates a transition to an anomalous state following a specific change point, demonstrating the event-level detection challenge.

Datasets

The benchmark evaluates 14 representative models across seven datasets:

  • Public Benchmarks: SWaT (water treatment), WADI (water distribution), SMD (server machines), SKAB (water circulation), and TEP (chemical process).
  • Industrial Data: Proprietary datasets from a Steam Turbine and a Nuclear Power Plant (NPP) Turbogenerator.

The unified application of event-level labels and stress augmentations across these diverse sources establishes a new standard for AD benchmarking.

Key Insights and Results

The benchmarking process reveals that standard "clean data" leaderboards are often misleading. Key findings include:

  • No Universal Winner: Model rankings fluctuate significantly under different stress profiles. For instance, while flow-based models perform well on stationary data, they are highly fragile to monotonic drift.
  • Robustness of Graph Models: Graph-structured models (e.g., GDN, GBAD) demonstrate superior robustness to sensor dropouts and long-duration events. On the SWaT dataset under additive noise, a standard graph autoencoder (GBAD) saw its F1 score drop from 0.804 to 0.677, whereas a hybrid graph-attention model (MTAD-GAT) maintained stability, dropping only from 0.762 to 0.756.
  • Impact of Drift: Log drift caused catastrophic failure in flow-based models on SKAB and NPP datasets. In an ablation of the GANF model, replacing normalizing flows with a simpler Gaussian density estimator reduced high-stress F1 on SWaT from ~0.75 to ~0.57.
  • Toxic Sensor Mitigation: Identifying and disabling a single "toxic" sensor via zero-channel probing improved the GBAD model's F1 score on the Turbogenerator dataset by 54% (0.38 to 0.58).
  • Inductive Bias vs. Robustness: Simplifications intended to boost speed can compromise reliability. Fixing a learned graph in the GANF model provided a minor clean-set gain (0.5–1.0 points) but increased drift sensitivity by approximately 8x.

Conclusion

Model selection for IoT anomaly detection must be a design choice informed by the expected operational stresses of the target system. While "SOTA" performance on clean datasets is a common goal, robustness to drift, noise, and sensor failure is paramount for safety-critical applications. Future work will focus on embedding data-specific stress testing to further refine these evaluations for plant-specific failure modes.

Summary updated

Abstract

Anomaly detection (AD) for safety-critical IoT time series should be judged at the event level: reliability and earliness under realistic perturbations. Yet many studies still emphasize point-level results on curated base datasets, limiting value for model selection in practice. We introduce an evaluation protocol with unified event-level augmentations that simulate real-world issues: calibrated sensor dropout, linear and log drift, additive noise, and window shifts. We also perform sensor-level probing via mask-as-missing zeroing with per-channel influence estimation to support root-cause analysis. We evaluate 14 representative models on five public anomaly datasets (SWaT, WADI, SMD, SKAB, TEP) and two industrial datasets (steam turbine, nuclear turbogenerator) using unified splits and event aggregation. There is no universal winner: graph-structured models transfer best under dropout and long events (e.g., on SWaT under additive noise F1 drops 0.804->0.677 for a graph autoencoder, 0.759->0.680 for a graph-attention variant, and 0.762->0.756 for a hybrid graph attention model); density/flow models work well on clean stationary plants but can be fragile to monotone drift; spectral CNNs lead when periodicity is strong; reconstruction autoencoders become competitive after basic sensor vetting; predictive/hybrid dynamics help when faults break temporal dependencies but remain window-sensitive. The protocol also informs design choices: on SWaT under log drift, replacing normalizing flows with Gaussian density reduces high-stress F1 from ~0.75 to ~0.57, and fixing a learned DAG gives a small clean-set gain (~0.5-1.0 points) but increases drift sensitivity by ~8x.
BibTeX
@misc{zhevnenko2026benchmarking,
  author = {Dmitry Zhevnenko and Ilya Makarov and Aleksandr Kovalenko and Fedor Meshchaninov and Anton Kozhukhov and Vladislav Travnikov and Makar Ippolitov and Kirill Yashunin and Iurii Katser},
  title = {{Benchmarking IoT Time-Series AD with Event-Level Augmentations}},
  year = {2026},
  eprint = {2602.15457},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.15457},
}
Figure