Who Watches the Watchers: A Multi-Task Benchmark for Anomaly Detection

Phil Demetriou, Ingolf Becker, Stephen HailesICISSP 2022 · DOI

TL;DR

The paper introduces "Aftershock," a novel multi-task benchmark designed to standardize the evaluation of anomaly detection systems, particularly for IoT and cyber-physical systems. By providing a common set of datasets and evaluation metrics, Aftershock aims to drive the development of more generalizable and practically useful anomaly detection models.

What Problem Does This Paper Address?

The proliferation of interconnected systems, especially in the Internet of Things (IoT) and cyber-physical domains, has created a critical need for robust anomaly detection. These systems are often resource-constrained and vulnerable to both component failures and malicious attacks. However, the research landscape for anomaly detection is fragmented, lacking a standardized basis for evaluating different methods. Researchers typically report results on isolated scenarios, making it difficult to compare approaches, understand their generalizability, or choose the best solution for a specific context. This lack of a common benchmark hinders progress and makes it challenging for practitioners to select and deploy effective anomaly detection systems.

Methods

The core contribution is the creation of Aftershock, a multi-task benchmark. The authors adapted and standardized existing public datasets into anomaly detection-specific benchmarks. They then applied a diverse set of anomaly detection algorithms to these benchmarks to establish performance baselines. Aftershock emphasizes reproducibility and fair comparison by using a containerization approach for model submission and evaluation. Models are trained on publicly available development datasets and then evaluated on privately accessible test datasets within an air-gapped infrastructure to prevent data leakage. This process ensures that all models are tested under identical conditions.

Datasets

Aftershock features ten anomaly detection tasks derived from publicly accessible sources, covering a range of feature counts, data quantities, and difficulties. These datasets are categorized into:

  • Cyber-Physical Systems: Four datasets, including variants of the Tennessee-Eastman (TE) industrial process (Fortran, Matlab implementations, and raw data) and the SWaT (Secure Water Treatment) dataset.
  • Computer Networks: Three datasets, including CIC-IDS-2017 and two instances of the legacy KDD Cup 1999 dataset (HTTP and SMTP observations).
  • Cross-domain: Three datasets selected for their indication of cross-task generality: Shuttle (Statlog), CoverType (ForestCover), and Mammography.

These datasets are split into publicly accessible development sets for training and privately accessible test sets for evaluation.

Key Insights and Ideas

  • The Need for Standardization: The paper strongly highlights the critical lack of standardized benchmarks in anomaly detection, which impedes scientific progress and practical adoption.
  • Generalizability is Key: Aftershock is designed to reward models that perform well across a variety of tasks, pushing research beyond highly specialized solutions.
  • Temporal vs. Non-Temporal Tasks: The results suggest that temporal anomaly detection tasks are generally more challenging than non-temporal ones, indicating a need for more sophisticated temporal methods.
  • Performance vs. Runtime: The study found that inference runtime is not always correlated with better performance, and some highly efficient algorithms (like Isolation Forest) can achieve competitive results.
  • "Off-the-Shelf" Limitations: Even with standard algorithms, some anomalies, particularly within the Tennessee-Eastman dataset, remain difficult to detect, underscoring that anomaly detection is far from a solved problem.

Key Results

The paper presents results for several "off-the-shelf" anomaly detection models (Isolation Forest, One-Class SVM, Minimum Covariance Determinant, Local Outlier Factor) and two additional algorithms (MStream, Bionic).

  • AUC Scores: Most models demonstrated better-than-random performance across tasks. The geometric mean AUC score across all attempted tasks was 0.805.
  • Temporal Task Performance: Temporal tasks generally showed lower AUC scores compared to non-temporal tasks.
  • Inference Runtimes: Runtimes varied significantly, with some models (e.g., Isolation Forest, Bionic) exhibiting very low per-record inference times (around 0.001-0.002 ms/record), while others like One-Class SVMs were often too slow to complete evaluation within the given resource constraints.
  • MStream Variance: MStream showed higher variance in AUC scores across tasks, particularly excelling in temporal tasks.

Limitations and Future Work

  • Dataset Limitations: While efforts were made to adapt existing datasets, some may still carry risks of inference due to their origins. The authors hope to include more proprietary tasks in the future.
  • Model Submission Constraints: While containerization allows for diverse models, the current infrastructure has resource limitations (4 CPU cores, 4GB RAM, 24 aggregate core hours) which can lead to submissions being unranked.
  • Future Work: The authors plan to expand the set of available tasks, engage more academic and industry partners, and explore options for hardware-in-the-loop evaluation and support for fully private datasets. They also suggest a stronger research focus on temporal anomaly detection methods.

Summary updated

BibTeX
@inproceedings{demetriou2022who,
  author = {Phil Demetriou and Ingolf Becker and Stephen Hailes},
  title = {{Who Watches the Watchers: A Multi-Task Benchmark for Anomaly Detection}},
  booktitle = {ICISSP},
  year = {2022},
  doi = {10.5220/0010915000003120},
}
Figure