DNN-GDITD: Out-of-distribution detection via Deep Neural Network based Gaussian Descriptor for Imbalanced Tabular Data

Priyanka Chudasama, Anil Surisetty, Aakarsh Malhotra, Alok SingharXiv 2024 · OpenReview

TL;DR

This paper introduces DNN-GDITD, a novel method for detecting out-of-distribution (OOD) samples in imbalanced tabular datasets. By transforming the latent space into Gaussian-like clusters and employing a combination of specialized loss functions, DNN-GDITD achieves robust OOD detection and improved classification accuracy, outperforming existing methods.

What Problem Does This Paper Address?

Many real-world applications, such as finance and autonomous driving, deal with imbalanced datasets where some classes have significantly fewer samples than others. This imbalance poses a challenge for standard classification models, making it difficult to correctly identify rare events. Furthermore, these systems must also be able to detect "out-of-distribution" (OOD) samples – data points that come from a distribution not seen during training. When combined with class imbalance, OOD detection becomes even more critical, as OOD samples can be mistakenly classified as minority classes, leading to potentially severe consequences. Existing OOD detection methods often struggle in these imbalanced tabular data scenarios.

Methods

The core of DNN-GDITD is its approach to modeling the latent space representations of data. It aims to transform the latent space into a set of $k$ independent Gaussian-like clusters, where $k$ is the number of known classes. Each cluster is characterized by a mean ($\mu$) and standard deviation ($\sigma$).

To achieve this, DNN-GDITD utilizes a novel combination of four loss functions:

  • Pull Loss: This loss encourages data points to be close to the mean of their true class cluster while simultaneously making the clusters more compact.
  • Score-based Loss (SL): This loss ensures that the "score" (related to the distance from a cluster) for a data point's true class is non-negative, while scores for other classes are negative. This helps in clearly separating in-distribution (ID) samples from OOD samples.
  • Effective Focal Loss (EFL1 and EFL2): These losses are adapted from focal loss, which is effective in handling class imbalance. EFL1 uses the distance-based measure, and EFL2 uses the score-based measure to focus on difficult-to-classify samples.

These losses are combined into a net loss function that is optimized to train a base DNN and learn the cluster parameters ($\mu_i, \sigma_i$). For OOD detection during testing, if a sample's scores for all known classes are negative, it's classified as OOD. Otherwise, it's assigned to the class with the highest score.

Datasets

The proposed DNN-GDITD was evaluated on four tabular datasets:

  • Synthetic Financial Dispute Dataset: A custom dataset created from a fraud dataset using SMOTE, designed to represent highly imbalanced scenarios.
  • Gas Sensor [8]: A publicly available dataset.
  • Drive Diagnosis [11]: A publicly available dataset.
  • MNIST [5]: A publicly available dataset (used in a tabular format).

These datasets were chosen to cover a range of imbalance scenarios, from highly imbalanced to balanced. The authors also introduced a "Minority-class Down-Sampling Ratio" (MDSR) to systematically control the level of imbalance during experiments.

Key Insights and Ideas

  • Spherical Decision Boundaries for OOD: The paper argues that using spherical decision boundaries in the latent space, as opposed to linear ones (like in standard softmax), is more effective for distinguishing OOD samples. This is because it allows for a clearer separation based on distance and radius.
  • The Synergy of Distance and Score: The combination of distance-based losses (Pull, EFL1) and score-based losses (SL, EFL2) is crucial. While distance-based losses help form compact clusters, score-based losses are essential for explicitly defining the OOD boundary.
  • Handling Imbalance is Key: The effective focal loss component is vital for ensuring that the model performs well even when minority classes have very few samples.
  • Robustness to Imbalance: DNN-GDITD shows consistent performance across various levels of data imbalance, a significant advantage over methods that degrade more sharply.

Key Results

  • Superior OOD Detection: DNN-GDITD consistently outperformed three baseline methods (Softmax, Mahalanobis, and Deep-MCDD) in OOD detection across various metrics (TNR, AUROC, AUPR), showing an average boost of 3.32%.
  • Comparable ID Classification: While not always achieving the absolute highest ID class accuracy (Softmax sometimes performed slightly better), DNN-GDITD's performance was comparable and demonstrated a better overall balance across ID, OOD, and minority class performance.
  • Statistical Significance: The improvements in OOD detection metrics were statistically significant, as confirmed by a Wilcoxon signed-rank test.
  • Ablation Study Highlights Loss Importance: An ablation study showed that omitting any of the four proposed loss functions led to a performance drop, underscoring the effectiveness of the combined approach. The score-based loss was particularly highlighted for its role in ensuring correct OOD classification.

Limitations and Future Work

  • Tabular Data Focus: The current work is specifically tailored for tabular data. The authors suggest that DNN-GDITD could potentially be extended to other domains and modalities, which would be a valuable future direction.
  • Outlier Presence: While the method is designed for OOD detection, the paper acknowledges that outliers can be present in various domains, implying that further research might be needed to fully address all types of anomalous data.
  • Computational Cost: The paper mentions that updating DNN parameters and cluster parameters simultaneously can be challenging with standard gradient descent, leading to the use of Block Coordinate Descent (BCD). Further optimization of the training process could be explored.

Summary updated

Abstract

Classification tasks present challenges due to class imbalances and evolving data distributions. Addressing these issues requires a robust method to handle imbalances while effectively detecting out-of-distribution (OOD) samples not encountered during training. This study introduces a novel OOD detection algorithm designed for tabular datasets, titled Deep Neural Network-based Gaussian Descriptor for Imbalanced Tabular Data (DNN-GDITD). The DNN-GDITD algorithm can be placed on top of any DNN to facilitate better classification of imbalanced data and OOD detection using spherical decision boundaries. Using a combination of Push, Score-based, and focal losses, DNN-GDITD assigns confidence scores to test data points, categorizing them as known classes or as an OOD sample. Extensive experimentation on tabular datasets demonstrates the effectiveness of DNN-GDITD compared to three OOD algorithms. Evaluation encompasses imbalanced and balanced scenarios on diverse tabular datasets, including a synthetic financial dispute dataset and publicly available tabular datasets like Gas Sensor, Drive Diagnosis, and MNIST, showcasing DNN-GDITD's versatility.
BibTeX
@misc{chudasama2024dnn,
  author = {Priyanka Chudasama and Anil Surisetty and Aakarsh Malhotra and Alok Singh},
  title = {{DNN-GDITD: Out-of-distribution detection via Deep Neural Network based Gaussian Descriptor for Imbalanced Tabular Data}},
  year = {2024},
  eprint = {2409.00980},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.00980},
}
Figure