Towards Lightweight, Model-Agnostic and Diversity-Aware Active Anomaly Detection

Xu Zhang, Yuan Zhao, Ziang Cui, Liqun Li, Shilin He, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, Dongmei ZhangICLR 2023 · OpenReview

Executive Summary

LMADA is a novel framework for Active Anomaly Detection (AAD) designed to be lightweight, compatible with any existing anomaly detector, and capable of discovering a wider variety of anomalies. By employing a diversity-aware sampling strategy and an innovative "proxy model" tuning approach, LMADA achieves a 74% average F1-score improvement over standard unsupervised methods, significantly outperforming current state-of-the-art techniques.

The Challenge of Unsupervised Anomaly Detection

Anomaly detection is traditionally an unsupervised task: a model is fed data and flags instances that appear anomalous. However, a model's mathematical definition of an "anomaly" does not always align with what a human analyst actually cares about.

Active Anomaly Detection (AAD) bridges this gap by introducing a human-in-the-loop. It allows an analyst to label a small subset of samples (e.g., confirming a real cyberattack or dismissing a routine system update) to iteratively refine the model's accuracy.

Active Anomaly Detection Pipeline

Despite its promise, the authors identify three major flaws in current AAD methodologies:

  1. The "Top-Selection" Trap: Most models only query humans to label the samples with the absolute highest anomaly scores. If a detector initially misses a specific class of anomaly, those samples will never be surfaced for labeling, and the model will never learn to identify them.
  2. Model Dependency: Existing AAD tools are often "bespoke"—hard-coded to optimize one specific type of detector (like an Isolation Forest). Switching to a different detection algorithm requires rewriting the entire AAD logic.
  3. Computational Heaviness: Many methods become prohibitively slow or memory-intensive as datasets scale, rendering them impractical for real-world, high-volume environments.

The LMADA Framework

To overcome these limitations, the researchers propose LMADA, a framework built on two core components:

1. Diversity-Aware Sample Selector

Instead of exclusively picking top-ranked anomalies, LMADA utilizes Determinantal Point Processes (DPP). This acts as a "repulsion" mechanism during sample selection. It searches for samples that have high anomaly scores but are also distinct from one another and from previously labeled data. This ensures the human analyst reviews a diverse spectrum of potential issues, enabling the model to discover "hidden" anomaly types it initially overlooked.

2. Model-Agnostic Tuner

To ensure compatibility with any base detector, LMADA employs a clever two-step "proxy" approach:

  • Proxy Approximation: A simple neural network (the proxy) is trained to mimic the behavior of the "black-box" base detector.
  • Representation Adjuster: When human feedback is received, the base detector itself is left untouched. Instead, LMADA trains a lightweight "adjuster" layer that transforms the input data. This effectively "warps" the feature space so that the unmodified base detector naturally assigns higher scores to the types of anomalies the human just labeled.

Key Insights & Innovations

  • The "Smurf" Discovery: The danger of the "Top-Selection" trap is perfectly illustrated in the KDD99-SA dataset. A specific type of network attack called "smurf" accounts for 71% of the dataset's anomalies, yet standard detectors initially assign them relatively low anomaly scores.
Anomaly Score Distributions

Traditional AAD methods remain blind to these attacks because they only sample the highest-scoring points. LMADA’s diversity-aware sampling caught the "smurf" attacks in its very first iteration, resulting in a massive leap in detection performance.

  • Input Warping vs. Model Retraining: A major conceptual breakthrough in LMADA is realizing that you do not need to retrain a complex, rigid base model to incorporate human feedback. By applying a non-linear transformation to the input features via the Representation Adjuster, you can effectively steer any existing detector toward the user's specific interests.
  • Efficiency through Truncation: To prevent the complex DPP mathematics from bottlenecking performance on massive datasets, LMADA only performs diversity sampling on the top $\alpha\%$ of samples. This significantly reduces computational overhead without sacrificing the quality of the selected samples.

Performance Highlights

The researchers evaluated LMADA across 8 public benchmark datasets spanning network security, medical diagnostics, and physical systems (ranging from 1.8K to 286K samples).

  • Massive Accuracy Gains: LMADA achieved an average 74% improvement in F1-Score across the evaluated datasets.
  • State-of-the-Art Superiority: It consistently outperformed competing AAD methods like Meta-AAD and FIF, particularly on datasets where anomalies were initially obscured.
  • True Agnosticism: LMADA successfully boosted the performance of five entirely different base detectors (Isolation Forest, PCA, OCSVM, LODA, and AutoEncoders), proving its universal applicability.

Limitations and Future Directions

While highly effective, LMADA has a few constraints:

  • Proxy Accuracy: The framework relies on the proxy neural network accurately mimicking the base detector. If the base detector is exceptionally complex or highly non-continuous, the proxy may struggle to approximate it perfectly.
  • Hyperparameter Sensitivity: The paper suggests a default "pre-truncation" rate of 10% for the DPP sampling. However, the optimal balance between diversity and raw anomaly score may require tuning based on the specific contamination level of a new dataset.
  • Human-in-the-Loop Assumptions: Like all AAD systems, LMADA assumes the human analyst provides flawless labels. Future iterations could explore mechanisms to handle "noisy" or conflicting feedback from multiple analysts.

Summary updated

Abstract

Active Anomaly Discovery (AAD) is flourishing in the anomaly detection research area, which aims to incorporate analysts’ feedback into unsupervised anomaly detectors. However, existing AAD approaches usually prioritize the samples with the highest anomaly scores for user labeling, which hinders the exploration of anomalies that were initially ranked lower. Besides, most existing AAD approaches are specially tailored for a certain unsupervised detector, making it difficult to extend to other detection models. To tackle these problems, we propose a lightweight, model-agnostic and diversity-aware AAD method, named LMADA. In LMADA, we design a diversity-aware sample selector powered by Determinantal Point Process (DPP). It considers the diversity of samples in addition to their anomaly scores for feedback querying. Furthermore, we propose a model-agnostic tuner. It approximates diverse unsupervised detectors with a unified proxy model, based on which the feedback information is incorporated by a lightweight non-linear representation adjuster. Through extensive experiments on 8 public datasets, LMADA achieved 74% F1-Score improvement on average, outperforming other comparative AAD approaches. Besides, LMADA can also achieve significant performance boosting under any unsupervised detectors.
BibTeX
@inproceedings{zhang2023towards,
  author = {Xu Zhang and Yuan Zhao and Ziang Cui and Liqun Li and Shilin He and Qingwei Lin and Yingnong Dang and Saravan Rajmohan and Dongmei Zhang},
  title = {{Towards Lightweight, Model-Agnostic and Diversity-Aware Active Anomaly Detection}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=-vKlt84fHs},
}
Figure