Generative Imputation and Stochastic Prediction

Mohammad Kachuee, Kimmo Kärkkäinen, Orpaz Goldstein, Sajad Darabi, Majid Sarrafzadeh

TL;DR

This paper introduces a novel method called Generative Imputation and Stochastic Prediction (GI) that tackles incomplete datasets by not only filling in missing values but also quantifying the uncertainty associated with these imputations and their impact on classification predictions. This approach is crucial for real-world applications where understanding prediction confidence is as important as accuracy.

What Problem Does This Paper Address?

Many machine learning applications deal with incomplete datasets, where some feature values are missing. Traditional imputation methods focus on filling these gaps with plausible values. However, this process inherently introduces uncertainty. Simply replacing missing values can mask this uncertainty, leading to overconfident predictions and a false sense of accuracy. The paper addresses the limitation of existing methods by proposing a framework that explicitly models and leverages this uncertainty, providing not just imputed data but also a measure of confidence in the resulting predictions. This is particularly important in domains like healthcare, where understanding the certainty of a diagnosis is critical.

Methods

The proposed method, Generative Imputation and Stochastic Prediction (GI), consists of two main stages:

  1. Generative Imputation: This stage uses a Generative Adversarial Network (GAN) framework. A generator network is trained to produce imputations for missing features, conditioned on the observed features and a random noise vector. A discriminator network is trained to distinguish between real and imputed features. This adversarial process encourages the generator to produce realistic imputations that capture the underlying data distribution. Unlike some GAN-based methods, GI avoids using an MSE loss term in the discriminator, which can lead to mode collapse and biased imputations.
  2. Stochastic Prediction: After imputation, a predictor network is trained. Instead of training on a single imputed dataset (like traditional multiple imputation), the predictor is trained on multiple samples generated by the imputer for each instance. This allows the predictor to learn from the distribution of possible imputations and, consequently, to estimate the uncertainty in class assignments. The method leverages Monte Carlo estimation to approximate the distribution of target classes given the observed features and the distribution of missing features.

Datasets

The authors evaluated their method on a variety of datasets:

  • Image Datasets: CIFAR-10 and MNIST.
  • Tabular Datasets: UCI Landsat, MIT-BIH arrhythmia, Diabetes, Cholesterol, and Hypertension classification datasets.

The paper does not introduce new benchmark datasets but rather uses established ones to demonstrate the effectiveness of their approach across different data modalities and missingness patterns.

Key Insights and Ideas

  • Uncertainty is Key: The core insight is that missing data inherently introduces uncertainty, not just in the imputed values but also in the final predictions. Simply filling in values can be misleading.
  • Stochastic Prediction for Confidence: Training a predictor on multiple imputed samples allows it to learn the distribution of possible outcomes, enabling it to provide meaningful confidence estimates for its predictions.
  • GANs for Realistic Imputation: Using a GAN framework for imputation helps generate more realistic and diverse imputations compared to methods that might rely on simpler statistical models or MSE-based losses, which can lead to mode collapse.
  • Avoiding MSE Bias: The decision to omit an MSE loss term in the GAN discriminator is crucial for preventing bias towards the mean of the distribution and allowing for a more faithful representation of the conditional distribution of missing features.

Key Results

  • Superior Classification Accuracy: GI consistently outperformed other state-of-the-art methods (MisGAN, GAIN, DAE, MICE) in terms of top-1 classification accuracy across various datasets and missingness rates, particularly for higher missing rates. For example, on CIFAR-10, GI achieved the best results in 5 out of 6 tested scenarios.
  • Better Calibration: The accuracy versus certainty plots (Figure 4) show that GI's confidence estimates are better calibrated, meaning its reported confidence levels align more closely with actual prediction accuracy, compared to other methods.
  • Realistic Imputations: FID scores on CIFAR-10 demonstrated that GI generates higher quality imputations, especially for uniform missingness structures.
  • Robustness to Missingness Mechanisms: GI showed good performance across different missingness mechanisms (uniform, rectangular, foreground/background) on MNIST, indicating its robustness.

Limitations and Future Work

  • Computational Cost: While the method is effective, training GANs and performing multiple imputations for stochastic prediction can be computationally intensive.
  • MAR/MNAR: The experiments primarily focused on MCAR (Missing Completely At Random) mechanisms. While the authors state their method is compatible with MAR (Missing At Random), further investigation into MNAR (Missing Not At Random) scenarios would be valuable.
  • Hyperparameter Tuning: Like most deep learning methods, GI likely requires careful hyperparameter tuning for optimal performance.
  • Advanced Encoding: For tabular data, the paper used a simple one-hot encoding with added noise. More advanced encoding methods could potentially further improve performance.

The authors suggest that more advanced encoding methods could be explored for tabular data and that their framework is readily compatible with MAR structures, hinting at future work in these areas.

Summary updated

Abstract

In many machine learning applications, we are faced with incomplete datasets. In the literature, missing data imputation techniques have been mostly concerned with filling missing values. However, the existence of missing values is synonymous with uncertainties not only over the distribution of missing values but also over target class assignments that require careful consideration. In this paper, we propose a simple and effective method for imputing missing features and estimating the distribution of target assignments given incomplete data. In order to make imputations, we train a simple and effective generator network to generate imputations that a discriminator network is tasked to distinguish. Following this, a predictor network is trained using the imputed samples from the generator network to capture the classification uncertainties and make predictions accordingly. The proposed method is evaluated on CIFAR-10 and MNIST image datasets as well as five real-world tabular classification datasets, under different missingness rates and structures. Our experimental results show the effectiveness of the proposed method in generating imputations as well as providing estimates for the class uncertainties in a classification task when faced with missing values.
BibTeX
@article{kachuee2022generative,
  author = {Mohammad Kachuee and Kimmo Kärkkäinen and Orpaz Goldstein and Sajad Darabi and Majid Sarrafzadeh},
  title = {{Generative Imputation and Stochastic Prediction}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=27ECM0vwTn9},
}
Figure