Multilevel Anomaly Detection for Mixed Data

Kien Do, Truyen Tran, Svetha VenkatesharXiv 2016 · OpenReview

Abstract

Anomalies are those deviating from the norm. Unsupervised anomaly detection often translates to identifying low-density regions. Major problems arise when data is high-dimensional and mixed with discrete and continuous attributes. We propose MIXMAD (MIXed data Multilevel Anomaly Detection), an ensemble method that estimates sparse regions across multiple levels of abstraction. The hypothesis is that for domains where multiple data abstractions exist, a data point may be anomalous with respect to the raw representation or more abstract representations. To this end, our method sequentially constructs an ensemble of Deep Belief Nets (DBNs) with varying depths. Each DBN acts as an energy-based detector at a predefined abstraction level. At the base of each DBN, a Mixed-variate Restricted Boltzmann Machine (Mv.RBM) models the density of mixed data. Predictions across the ensemble are combined via rank aggregation. Evaluated on high-dimensional real-world datasets, MIXMAD demonstrates that multilevel abstraction is a sensible strategy and empirically superior to popular unsupervised detection methods for both homogeneous and mixed data.

Addressing the Challenges of Modern Data

Unsupervised anomaly detection typically relies on finding data points in low-density regions. However, this approach faces two significant hurdles in real-world applications:

  1. High Dimensionality: The "curse of dimensionality" causes traditional distance metrics, such as Euclidean distance, to become less meaningful as all points appear roughly equidistant.
  2. Mixed Data Types: Real-world datasets often combine continuous variables (e.g., temperature) and discrete categories (e.g., product types). Defining a unified mathematical distance or joint probability distribution for these disparate types is notoriously difficult.

Existing methods often address one issue while failing at the other. Distance-based methods like $k$-Nearest Neighbors ($k$-NN) struggle with high-dimensional mixed data, while many deep learning approaches rely on reconstruction error, which may focus on raw noise rather than underlying density.

The MIXMAD Methodology

MIXMAD is built on the Multilevel Anomaly Detection (MAD) hypothesis: a data point might appear normal at a raw feature level but reveal itself as highly anomalous at a higher, abstract semantic level (or vice versa).

The MIXMAD architecture sequentially builds an ensemble of DBNs of varying depths, aggregating their rankings to identify anomalies.

Core Components:

  • Mixed-variate RBM (Mv.RBM): At the base layer, this specialized neural network ingests mixed continuous and discrete data, transforming it into a homogeneous binary representation suitable for deeper processing.
  • Multilevel Abstraction: The model constructs an ensemble of Deep Belief Networks (DBNs) with varying depths (e.g., 1, 2, and 3 layers). Each successive layer learns a progressively higher-level abstraction of the data.
  • Free-Energy Scoring: Instead of reconstruction error, MIXMAD calculates the free-energy of the abstracted data at the top layer of each DBN. Free-energy serves as a computationally efficient proxy for anomaly scoring; higher free-energy indicates lower probability density.
  • Rank Aggregation: Since free-energy values are not directly comparable across different abstraction levels, the model ranks anomalies at each level. These rankings are then combined using a flexible $p$-norm aggregation technique to reach a final consensus.

Experimental Evaluation

The authors evaluated MIXMAD across diverse datasets, categorized into homogeneous (single type) and mixed data. Homogeneous tests included MNIST (digit '8' as normal, others as anomalies), InternetAds, and clinical Preterm Birth records. Mixed data tests utilized standard UCI benchmarks such as KDD99-10, Australian Credit, and Auto MPG.

Key Insights and Results

The experiments confirmed that anomalies hide at different semantic levels. In the MNIST dataset, for example, $k$-NN often flagged normal '8's as anomalies due to pixel-level variations like stroke thickness. In contrast, MIXMAD recognized the abstract concept of the digit '8', correctly identifying actual different digits as the true anomalies.

Visual comparison of anomaly detection on MNIST. MIXMAD-L2p2 achieves the lowest error count by correctly identifying abstract digit shapes rather than pixel-level noise.
  • Superior Performance: MIXMAD significantly outperformed shallow baselines. On MNIST, a 2-layer MIXMAD achieved an AUC of 0.897, compared to $k$-NN's 0.804. On the mixed-data Auto MPG set, it achieved a perfect F-score of 1.00.
  • Generation vs. Detection: The authors found that anomaly detection requires identifying "empty space" rather than memorizing every mode of normal data. By using smaller RBMs for the detection step, MIXMAD avoids creating overly complex energy wells that lead to false alarms.
  • Optimal Depth: Ensembles combining 2 or 3 layers generally yielded the best results. However, going "too deep" can be counterproductive, as representations may become too flat to distinguish density regions effectively.

Conclusion and Future Directions

MIXMAD demonstrates that multilevel abstraction is a powerful tool for high-dimensional, mixed-data anomaly detection. While the current $p$-norm rank aggregation is effective, future work could explore more sophisticated aggregation methods. Additionally, the authors suggest extending this framework to multimodal data, such as integrating images with text tags, to model different modalities at varying abstraction levels.

Summary updated

Abstract

Anomalies are those deviating from the norm. Unsupervised anomaly detection often translates to identifying low density regions. Major problems arise when data is high-dimensional and mixed of discrete and continuous attributes. We propose MIXMAD, which stands for MIXed data Multilevel Anomaly Detection, an ensemble method that estimates the sparse regions across multiple levels of abstraction of mixed data. The hypothesis is for domains where multiple data abstractions exist, a data point may be anomalous with respect to the raw representation or more abstract representations. To this end, our method sequentially constructs an ensemble of Deep Belief Nets (DBNs) with varying depths. Each DBN is an energy-based detector at a predefined abstraction level. At the bottom level of each DBN, there is a Mixed-variate Restricted Boltzmann Machine that models the density of mixed data. Predictions across the ensemble are finally combined via rank aggregation. The proposed MIXMAD is evaluated on high-dimensional realworld datasets of different characteristics. The results demonstrate that for anomaly detection, (a) multilevel abstraction of high-dimensional and mixed data is a sensible strategy, and (b) empirically, MIXMAD is superior to popular unsupervised detection methods for both homogeneous and mixed data.
BibTeX
@misc{do2016multilevel,
  author = {Kien Do and Truyen Tran and Svetha Venkatesh},
  title = {{Multilevel Anomaly Detection for Mixed Data}},
  year = {2016},
  eprint = {1610.06249},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1610.06249},
}
Figure