Introduction
Variational Autoencoders (VAEs) have emerged as a cornerstone of generative modeling, offering a robust framework for learning latent representations of complex data. However, their performance often degrades significantly when applied to real-world datasets characterized by missing values—a pervasive issue resulting from sensor failures, skipped survey responses, or data corruption.
The fundamental challenge identified in this research is posterior inconsistency. In an ideal Bayesian setting, the posterior distribution inferred from a subset of features should be mathematically consistent with the posterior derived from a larger set containing those features. Standard VAE architectures do not enforce this property. Consequently, encoders often learn "shortcuts" specific to certain missingness patterns, leading to biased latent distributions, overconfident yet inaccurate imputations, and poor performance in downstream tasks.
Methodology: Enforcing Posterior Consistency
To address this, we introduce a formal framework for posterior consistency in VAEs trained on incomplete data. Our approach shifts the focus from architectural modifications to the training objective itself.
Formal Definition and Regularization
We define posterior consistency based on first principles: the posterior distribution of a latent variable $z$ given a subset of features $x_P$ should relate predictably to the posterior given a larger set $x_Q$ (where $P \subset Q$). We derive a differentiable proxy for this consistency that can be integrated directly into the standard Evidence Lower Bound (ELBO) objective. This novel regularization term penalizes the model when the latent distribution generated from a limited feature set contradicts the distribution generated from a more complete set.
Training with Artificial Missingness
Our training strategy employs "artificial missingness" to robustify the encoder. During each iteration, the model is presented with an already incomplete data point. We then hide additional features and force the encoder to maintain consistency across these varying levels of information. This process ensures the latent space remains well-organized regardless of the specific missingness pattern encountered at test time.
Architectural Agnostic Approach
A key strength of this method is its orthogonality to existing VAE architectures. It can be seamlessly integrated into standard VAEs, permutation-invariant models like VAE-PNP, and advanced frameworks utilizing Normalizing Flows for high-dimensional density estimation.
Experimental Evaluation
We evaluated the proposed REG-VAE framework across a diverse suite of benchmarks from the UCI Machine Learning Repository, including datasets such as Boston Housing, Wine, and Breast Cancer. Testing was conducted under both Missing Completely at Random (MCAR) and Missing Not at Random (MNAR) conditions.
Quantitative Performance
Across nearly all datasets and missingness rates (ranging from 30% to 70%), REG-VAEs consistently outperformed non-regularized baselines. Key metrics included:
- Imputation Accuracy: Significant reductions in Root Mean Square Error (RMSE) for missing value prediction.
- Generative Quality: Improved Negative Log-Likelihood (NLL) and ELBO values, indicating a superior fit to the underlying data distribution.
- Active Feature Acquisition: In "Information Curve" experiments, REG-VAEs achieved target accuracy with fewer acquired features, demonstrating a more informative latent space for decision-making.
Qualitative Success in Image Inpainting
The impact of posterior consistency is most visible in qualitative tasks like image inpainting on the MNIST dataset.
As shown in the figure above, when presented with a highly ambiguous partial observation (a small vertical stroke from an original '4'), standard models (VAE-PNP and AM-VAE-PNP) collapse to a single, incorrect guess (the digit '9'). In contrast, our regularized model (REG-VAE-PNP) produces a diverse set of plausible completions, including '1's, '9's, '7's, and '3's. This ability to capture true epistemic uncertainty is vital for safety-critical applications where overconfidence can lead to catastrophic failures.
Discussion and Future Work
By prioritizing consistency over specific network structures, this work demonstrates that the training objective is a critical lever for handling missing data. REG-VAEs not only provide better imputations but also offer more reliable uncertainty estimates, which are essential for active learning and feature selection.
While the current approach shows significant promise, it introduces additional computational overhead during training due to the calculation of multiple ELBO terms. Future research will focus on optimizing this training efficiency and exploring more complex, non-ignorable missingness mechanisms (MNAR) in high-dimensional settings.
Summary updated