Addressing Class Imbalance with Probabilistic Graphical Models and Variational Inference

Yujia Lou, Jie Liu, Yuan Sheng, Jiawei Wang, Yiwei Zhang, Yaokun Ren

Abstract

This study proposes a method for imbalanced data classification based on deep probabilistic graphical models (DPGMs) to solve the problem that traditional methods have insufficient learning ability for minority class samples. To address the classification bias caused by class imbalance, we introduce variational inference optimization probability modeling, which enables the model to adaptively adjust the representation ability of minority classes and combines the class-aware weight adjustment strategy to enhance the classifier's sensitivity to minority classes. In addition, we combine the adversarial learning mechanism to generate minority class samples in the latent space so that the model can better characterize the category boundary in the high-dimensional feature space. The experiment is evaluated on the Kaggle "Credit Card Fraud Detection" dataset and compared with a variety of advanced imbalanced classification methods (such as GAN-based sampling, BRF, XGBoost-Cost Sensitive, SAAD, HAN). The results show that the method in this study has achieved the best performance in AUC, Precision, Recall and F1-score indicators, effectively improving the recognition rate of minority classes and reducing the false alarm rate.

Introduction

In real-world applications such as financial fraud detection, medical diagnosis, and cybersecurity, datasets are often characterized by extreme class imbalance. For instance, in a typical credit card transaction dataset, fraudulent activities might represent less than 0.2% of the total volume. Standard machine learning models, which prioritize overall accuracy, often fail to detect these rare but critical events, effectively treating them as noise. While traditional methods like oversampling (e.g., SMOTE) or cost-sensitive learning exist, they often suffer from overfitting or require extensive manual tuning. This study proposes a robust alternative using deep probabilistic graphical models (DPGMs).

Methodology

The proposed framework leverages the generative power of DPGMs to capture the underlying structure of minority classes through a multi-layered approach.

Deep Probabilistic Graphical Models (DPGMs)

Unlike deterministic classifiers, DPGMs model the joint probability distribution of the data. By introducing latent variables, the model learns a compressed representation of the input features, allowing it to generalize better from sparse data.

The proposed probabilistic graphical model structure, illustrating the dependencies between latent variables and observed data.

Variational Inference and Adversarial Learning

To handle the computational complexity of posterior distributions, the authors employ Variational Inference (VI) to optimize the Evidence Lower Bound (ELBO). This ensures the model accurately reconstructs data while regularizing the latent space. To further refine the decision boundaries, a Latent Adversarial Learning mechanism is introduced. A discriminator operates in the latent space, forcing the generator to produce more distinct and separable feature representations for the minority class.

Category-Aware Weighting

A dynamic weighting strategy is applied to the loss function. By scaling the loss based on the ratio of majority samples to minority samples, the model is penalized more heavily for misclassifying minority samples, effectively increasing the "signal" from rare events during training.

Experimental Setup

The model was rigorously tested on the Kaggle Credit Card Fraud Detection dataset, which contains 284,807 transactions with a fraud ratio of approximately 1:577. The data includes 30 numerical features derived from PCA. The dataset was split into training (70%), validation (15%), and testing (15%) sets to ensure robust evaluation.

Results and Discussion

The DPGM-based approach demonstrated superior performance across all key metrics compared to industry-standard baselines such as GAN-based sampling, XGBoost (Cost-Sensitive), and advanced attention models like HAN.

Metric Ours (DPGM) HAN [26] XGBoost-CS [24]
AUC 0.941 0.927 0.903
Precision 0.822 0.806 0.779
Recall 0.785 0.768 0.735
F1-Score 0.803 0.786 0.757

A critical component of this success is the model's ability to create a well-separated latent space. As shown in the T-SNE visualization below, the model successfully clusters fraudulent transactions (red) away from legitimate ones (blue), despite the massive class disparity.

T-SNE visualization of the latent space after training, showing clear separation between fraudulent (red) and legitimate (blue) clusters.

Conclusion

By combining probabilistic modeling with adversarial learning and variational inference, this research provides a powerful tool for imbalanced classification. The method not only improves detection rates but also offers better uncertainty modeling compared to heuristic resampling. Future work will focus on reducing computational overhead and exploring federated learning to enhance privacy in cross-institutional fraud detection.

Summary updated

Abstract

This study proposes a method for imbalanced data classification based on deep probabilistic graphical models (DPGMs) to solve the problem that traditional methods have insufficient learning ability for minority class samples. To address the classification bias caused by class imbalance, we introduce variational inference optimization probability modeling, which enables the model to adaptively adjust the representation ability of minority classes and combines the class-aware weight adjustment strategy to enhance the classifier's sensitivity to minority classes. In addition, we combine the adversarial learning mechanism to generate minority class samples in the latent space so that the model can better characterize the category boundary in the high-dimensional feature space. The experiment is evaluated on the Kaggle "Credit Card Fraud Detection" dataset and compared with a variety of advanced imbalanced classification methods (such as GAN-based sampling, BRF, XGBoost-Cost Sensitive, SAAD, HAN). The results show that the method in this study has achieved the best performance in AUC, Precision, Recall and F1-score indicators, effectively improving the recognition rate of minority classes and reducing the false alarm rate. This method can be widely used in imbalanced classification tasks such as financial fraud detection, medical diagnosis, and anomaly detection, providing a new solution for related research.
BibTeX
@misc{lou2025addressing,
  author = {Yujia Lou and Jie Liu and Yuan Sheng and Jiawei Wang and Yiwei Zhang and Yaokun Ren},
  title = {{Addressing Class Imbalance with Probabilistic Graphical Models and Variational Inference}},
  year = {2025},
  eprint = {2504.05758},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.05758},
}
Figure