Modeling Heterogeneous Statistical Patterns in High-dimensional Data by Adversarial Distributions: An Unsupervised Generative Framework

Han Zhang, Wenhao Zheng, Charley Chen, Kevin Gao, Yao Hu, Ling Huang, Wei Xu

Abstract

Identifying complex clusters in high-dimensional data remains a challenge due to the curse of dimensionality and the presence of heterogeneous statistical patterns. We propose FIRD (Feature Independence and adveRsarial Distributions), an unsupervised generative framework that models these patterns by making distribution components "compete" to explain specific data behaviors. By assuming conditional feature independence and employing adversarial priors—sparse for synchronized behavior and smooth for random behavior—FIRD effectively isolates anomalies. Evaluations on large-scale e-commerce datasets and standard benchmarks show that FIRD significantly outperforms existing dense-block and unsupervised methods, maintaining robustness even as noise levels increase.

Introduction

In modern high-dimensional datasets, such as e-commerce user logs, different groups often exhibit distinct statistical signatures. For instance, fraudsters may synchronize on specific dimensions (like registration time or device ID) to maximize efficiency while appearing random on others (like IP address) to evade detection. Traditional unsupervised methods often fail to capture these local, group-specific patterns because they either struggle with the computational complexity of joint distributions or apply global feature selection that discards critical "smoking gun" dimensions.

FIRD addresses these issues by modeling local patterns through a specialized generative process, allowing for the identification of sophisticated anomalies without the need for labeled training data.

Methods

The FIRD framework is built upon two foundational principles: the feature independence assumption and the use of adversarial distributions.

Generative Framework

To handle high dimensionality, FIRD assumes that features are conditionally independent within a given group. This allows the model to scale linearly with the number of dimensions, making it suitable for industrial-scale applications. The generative process is captured in the graphical model shown below.

Graphical representation of the FIRD generative framework.

In this model, $d_n$ represents the cluster assignment for user $n$, while $f_{nm}$ indicates whether a specific feature $m$ is "important" for that cluster. The observed data $x_{nm}$ is then generated based on these latent variables.

Adversarial Distributions

The core innovation of FIRD is the "competition" between distribution components. Instead of a standard mixture model, FIRD utilizes pairs of distributions with opposing priors:

  • Sparse Distributions: Encouraged by Dirichlet-like priors to capture synchronized or fraudulent behavior where values are concentrated.
  • Smooth Distributions: Encouraged to be uniform or smooth to capture the background noise of normal, random behavior.

The model parameters are optimized using an Expectation-Maximization (EM) algorithm, supplemented by a noise reduction step that filters out outliers that do not conform to any learned pattern.

Key Results

The researchers evaluated FIRD against state-of-the-art (SOTA) methods, including dense-block detection algorithms (D-Cube, M-Zoom, M-Biz) and standard anomaly detectors (Isolation Forest, OCSVM).

Performance and Robustness

On a real-world e-commerce dataset with 30 dimensions, FIRD achieved an average AUC of 0.82, a substantial improvement over the 0.64 achieved by dense-block methods. A critical finding was FIRD's robustness to varying Normal-to-Fraudster Ratios (NFR). As shown in the Precision-Recall curves below, while the performance of competing methods plummeted as the proportion of normal users increased, FIRD remained remarkably stable.

Precision-Recall curves comparing FIRD against state-of-the-art methods across varying Normal-to-Fraudster ratios.

Anomaly Detection Benchmarks

FIRD was also tested on nine standard ODDS library datasets (e.g., cardio, satellite). It provided a 5% average AUC improvement over SOTA methods like Isolation Forests. Furthermore, the model's complexity is $O(NGM)$, ensuring linear scalability with respect to the number of data points ($N$), clusters ($G$), and features ($M$).

Conclusion

FIRD provides a robust, interpretable, and efficient framework for unsupervised anomaly detection in high-dimensional spaces. By forcing distribution components to specialize through adversarial priors, it successfully distinguishes between synchronized fraudulent activity and random background noise. Future work will explore extending the framework to continuous data spaces and investigating more diverse adversarial pairs to capture a wider array of behavioral nuances.

Summary updated

Abstract

Since the label collecting is prohibitive and time-consuming, unsupervised methods are preferred in applications such as fraud detection. Meanwhile, such applications usually require modeling the intrinsic clusters in high-dimensional data, which usually displays heterogeneous statistical patterns as the patterns of different clusters may appear in different dimensions. Existing methods propose to model the data clusters on selected dimensions, yet globally omitting any dimension may damage the pattern of certain clusters. To address the above issues, we propose a novel unsupervised generative framework called FIRD, which utilizes adversarial distributions to fit and disentangle the heterogeneous statistical patterns. When applying to discrete spaces, FIRD effectively distinguishes the synchronized fraudsters from normal users. Besides, FIRD also provides superior performance on anomaly detection datasets compared with SOTA anomaly detection methods (over 5% average AUC improvement). The significant experiment results on various datasets verify that the proposed method can better model the heterogeneous statistical patterns in high-dimensional data and benefit downstream applications.
BibTeX
@inproceedings{zhang2020modeling,
  author = {Han Zhang and Wenhao Zheng and Charley Chen and Kevin Gao and Yao Hu and Ling Huang and Wei Xu},
  title = {{Modeling Heterogeneous Statistical Patterns in High-dimensional Data by Adversarial Distributions: An Unsupervised Generative Framework}},
  booktitle = {The Web Conference},
  year = {2020},
  doi = {10.1145/3366423.3380213},
}
Figure