Abstract
Anomalies are those deviating from the norm. Unsupervised anomaly detection often translates to identifying low-density regions. Major problems arise when data is high-dimensional and mixed with discrete and continuous attributes. We propose MIXMAD (MIXed data Multilevel Anomaly Detection), an ensemble method that estimates sparse regions across multiple levels of abstraction. The hypothesis is that for domains where multiple data abstractions exist, a data point may be anomalous with respect to the raw representation or more abstract representations. To this end, our method sequentially constructs an ensemble of Deep Belief Nets (DBNs) with varying depths. Each DBN acts as an energy-based detector at a predefined abstraction level. At the base of each DBN, a Mixed-variate Restricted Boltzmann Machine (Mv.RBM) models the density of mixed data. Predictions across the ensemble are combined via rank aggregation. Evaluated on high-dimensional real-world datasets, MIXMAD demonstrates that multilevel abstraction is a sensible strategy and empirically superior to popular unsupervised detection methods for both homogeneous and mixed data.
Addressing the Challenges of Modern Data
Unsupervised anomaly detection typically relies on finding data points in low-density regions. However, this approach faces two significant hurdles in real-world applications:
- High Dimensionality: The "curse of dimensionality" causes traditional distance metrics, such as Euclidean distance, to become less meaningful as all points appear roughly equidistant.
- Mixed Data Types: Real-world datasets often combine continuous variables (e.g., temperature) and discrete categories (e.g., product types). Defining a unified mathematical distance or joint probability distribution for these disparate types is notoriously difficult.
Existing methods often address one issue while failing at the other. Distance-based methods like $k$-Nearest Neighbors ($k$-NN) struggle with high-dimensional mixed data, while many deep learning approaches rely on reconstruction error, which may focus on raw noise rather than underlying density.
The MIXMAD Methodology
MIXMAD is built on the Multilevel Anomaly Detection (MAD) hypothesis: a data point might appear normal at a raw feature level but reveal itself as highly anomalous at a higher, abstract semantic level (or vice versa).
Core Components:
- Mixed-variate RBM (Mv.RBM): At the base layer, this specialized neural network ingests mixed continuous and discrete data, transforming it into a homogeneous binary representation suitable for deeper processing.
- Multilevel Abstraction: The model constructs an ensemble of Deep Belief Networks (DBNs) with varying depths (e.g., 1, 2, and 3 layers). Each successive layer learns a progressively higher-level abstraction of the data.
- Free-Energy Scoring: Instead of reconstruction error, MIXMAD calculates the free-energy of the abstracted data at the top layer of each DBN. Free-energy serves as a computationally efficient proxy for anomaly scoring; higher free-energy indicates lower probability density.
- Rank Aggregation: Since free-energy values are not directly comparable across different abstraction levels, the model ranks anomalies at each level. These rankings are then combined using a flexible $p$-norm aggregation technique to reach a final consensus.
Experimental Evaluation
The authors evaluated MIXMAD across diverse datasets, categorized into homogeneous (single type) and mixed data. Homogeneous tests included MNIST (digit '8' as normal, others as anomalies), InternetAds, and clinical Preterm Birth records. Mixed data tests utilized standard UCI benchmarks such as KDD99-10, Australian Credit, and Auto MPG.
Key Insights and Results
The experiments confirmed that anomalies hide at different semantic levels. In the MNIST dataset, for example, $k$-NN often flagged normal '8's as anomalies due to pixel-level variations like stroke thickness. In contrast, MIXMAD recognized the abstract concept of the digit '8', correctly identifying actual different digits as the true anomalies.
- Superior Performance: MIXMAD significantly outperformed shallow baselines. On MNIST, a 2-layer MIXMAD achieved an AUC of 0.897, compared to $k$-NN's 0.804. On the mixed-data Auto MPG set, it achieved a perfect F-score of 1.00.
- Generation vs. Detection: The authors found that anomaly detection requires identifying "empty space" rather than memorizing every mode of normal data. By using smaller RBMs for the detection step, MIXMAD avoids creating overly complex energy wells that lead to false alarms.
- Optimal Depth: Ensembles combining 2 or 3 layers generally yielded the best results. However, going "too deep" can be counterproductive, as representations may become too flat to distinguish density regions effectively.
Conclusion and Future Directions
MIXMAD demonstrates that multilevel abstraction is a powerful tool for high-dimensional, mixed-data anomaly detection. While the current $p$-norm rank aggregation is effective, future work could explore more sophisticated aggregation methods. Additionally, the authors suggest extending this framework to multimodal data, such as integrating images with text tags, to model different modalities at varying abstraction levels.
Summary updated