Real-time anomaly detection with superexperts

Raisa Dzhamtyrova, Carsten Maple

Introduction

Anomaly detection is a critical component in cybersecurity and industrial monitoring, yet practitioners often face significant challenges in real-world deployments. Most existing systems rely on a single unsupervised algorithm, which can fail when data patterns shift or attackers adapt. Furthermore, while theoretical models often assume immediate feedback, real-world ground-truth labels are frequently delayed by days or weeks. This paper introduces a "superexpert" framework that aggregates multiple unsupervised models and adapts to delayed feedback while maintaining strong theoretical performance guarantees.

Problem Statement

Two primary hurdles limit the effectiveness of current anomaly detection:

  1. Algorithm Volatility: No single model remains optimal indefinitely. A model effective for network intrusions might fail during baseline drifts.
  2. The Feedback Gap: Labels are rare and often arrive long after a prediction is made (e.g., after an analyst's weekly review).

Traditional "Prediction with Expert Advice" (PEA) frameworks typically require immediate feedback. This research extends these theories to handle non-stationary environments and significant feedback delays by allowing the ensemble to switch between experts dynamically.

Methodology

The core of the proposed system is the Aggregating Algorithm (AA), which maintains weights for a pool of "experts" (base anomaly detection models). The authors introduce several key innovations to handle real-world constraints:

  • AAP-current (Pack Averages): Adapts the AA for "packs" of outcomes that arrive after a delay, rather than updating on single immediate outcomes.
  • Fixed-share & Variable-share: These mechanisms allow the ensemble to "switch" its allegiance between experts. Fixed-share redistributes a portion of weights at each step, making it ideal for environments where the best model changes periodically. Variable-share is more targeted, sharing weights only from poorly performing experts.

The framework provides theoretical "regret bounds," ensuring the cumulative average loss remains close to that of the best possible sequence of experts, even with the handicap of delayed feedback.

Experimental Setup

The approach was evaluated using the Numenta Anomaly Benchmark (NAB), comprising 58 streaming data sources ranging from AWS server metrics to industrial sensors. The ensemble utilized 15 existing unsupervised anomaly detection algorithms from the NAB repository as base experts.

Results

The ensemble methods significantly outperformed standalone competitors on the NAB leaderboard across various metrics and delay scenarios.

AUC performance of ensemble methods vs. standalone algorithms across different delay intervals.

As shown in the AUC results (Figure 3), the best individual model (randomCutForest) achieved an AUC of 0.616. In contrast, the authors' Fixed-share approach reached an AUC of 0.998 with minimal delay and maintained a strong 0.893 even with a 100-step delay.

Performance metrics showing the impact of delay on various algorithms, illustrating the dominance of ensemble methods over standalone NAB algorithms.

Similar dominance was observed in other metrics (Figure 1), where ensemble methods consistently doubled or tripled the performance of standalone algorithms. The system's ability to maintain high accuracy despite delays of up to 100 data points demonstrates its practical utility in real-world monitoring scenarios.

Discussion and Conclusion

The "superexpert" concept proves that allowing an ensemble to switch between models dynamically provides superior performance over static selection. Beyond accuracy, the method offers interpretability; operators can monitor expert weights to understand which algorithms are currently trusted and why.

While the Fixed-share method is highly flexible, Variable-share is currently limited to bounded loss functions. Future work may focus on optimizing the switching rate ($\alpha$) and reducing the computational overhead of running multiple experts in parallel, which remains a concern for resource-constrained edge devices.

Summary updated

Abstract

The increasing connectivity of data and cyber-physical systems has resulted in a growing number of cyber-attacks. Real-time detection of such attacks, through the identification of anomalous activity, is required so that mitigation and contingent actions can be effectively and rapidly deployed. We propose a new approach for aggregating unsupervised anomaly detection algorithms and incorporating feedback when it becomes available. We apply this approach to open-source real datasets and show that both aggregating models, which we call experts, and incorporating feedback significantly improve the performance. An important property of the proposed approaches is their theoretical guarantees that they perform close to the best superexpert, which can switch between the best performing experts, in terms of the cumulative average losses.
BibTeX
@misc{dzhamtyrova2020real,
  author = {Raisa Dzhamtyrova and Carsten Maple},
  title = {{Real-time anomaly detection with superexperts}},
  year = {2020},
  eprint = {2010.03857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.03857},
}
Figure