Handling Incomplete Data with Joint Generative Models

43 concepts38 papers1 proposal

When building machine learning models for real-world applications—whether analyzing electronic health records, fusing multimodal sensor streams, or processing tabular data—missing values are an inevitable challenge. The traditional approach treats missing data as a preprocessing problem: we impute the missing features using heuristics like mean-filling, zero-padding, or nearest neighbors, and then pass the completed data to a downstream model.

This pipeline is fundamentally flawed. Ad-hoc imputation distorts the geometric structure of the data space and destroys complex feature correlations.

A more elegant and robust paradigm treats missing data handling not as an external preprocessing step, but as an intrinsic property of joint generative modeling. Instead of learning a direct mapping from inputs to outputs, these models learn the joint probability distribution of all variables, $P(X_1, X_2, \dots, X_d)$. When some features are missing, the model leverages the fundamental laws of probability—marginalization and conditioning—to handle the missingness naturally without ad-hoc modifications or task-specific retraining.

The Mechanics of Marginalization

How does a joint generative model handle missing inputs during inference? Suppose we want to evaluate the likelihood of an instance or classify it, but a subset of its features is missing. A standard discriminative model fails because its architecture expects a fixed-dimensional input vector. A generative model, however, can simply integrate out the unobserved dimensions.

Mathematically, if our features are split into observed variables $X_{\text{obs}}$ and missing variables $X_{\text{mis}}$, we can compute the marginal distribution by integrating over all possible values of the missing data:

$$P(X_{\text{obs}}) = \int P(X_{\text{obs}}, X_{\text{mis}}) \, dX_{\text{mis}}$$

Early foundations of this idea relied on mixture models. In A Mixture Model System for Medical and Machine Diagnosis, Stensmo et al. demonstrated how Gaussian Mixture Models (GMMs) could perform medical diagnosis by marginalizing over unobserved patient symptoms. To scale this to higher dimensions, Bach et al. introduced Thin Junction Trees to constrain model treewidth, ensuring that this marginalization step remains computationally tractable while preserving high-order dependencies.

In classification tasks, Mansinghka et al. showed in AClass: A simple, online, parallelizable algorithm for probabilistic classification that generative classifiers naturally outperform discriminative ones under high missingness because they compute $P(Y \mid X_{\text{obs}}) \propto P(X_{\text{obs}} \mid Y)P(Y)$ directly via marginalization, bypassing biased imputation techniques entirely. Similarly, Peharz et al. proved in The Most Generative Maximum Margin Bayesian Networks that integrating high generative likelihood into large-margin classifiers preserves this exact robustness at test time.

Modern architectures scale this capability to complex, heterogeneous data. For instance, in multimodal fusion where entire data streams (like audio or video) might be absent, Sidheekh et al. utilized Probabilistic Circuits in Credibility-Aware Multimodal Fusion Using Probabilistic Circuits to seamlessly integrate over missing modalities within a unified joint distribution. In non-linear dimensionality reduction, Lalchand et al. leveraged the marginalization property of Gaussian Process Latent Variable Models in Generalised GPLVM with Stochastic Variational Inference to ignore missing dimensions during likelihood calculations, allowing the model to reconstruct complex signals even when more than 90% of the features are missing.

This property is particularly valuable for edge computing hardware; Galindez Olascoaga et al. demonstrated in Towards Hardware-Aware Tractable Learning of Probabilistic Models that generative models facilitate hardware-aware inference by naturally handling missing features when sensors are deactivated to save power. Furthermore, Jaini et al. proved in Deep Homogeneous Mixture Models: Representation, Separation, and Approximation that deep hierarchical probabilistic structures are exponentially more efficient and accurate at capturing the global dependencies needed for robust data reconstruction than shallow ones.

Arbitrary Conditioning and Generative Imputation

When explicit reconstruction of the missing values is required, joint generative models provide a principled way to sample from the exact conditional distribution $P(X_{\text{mis}} \mid X_{\text{obs}})$. Because the model captures the global dependencies of the data, the imputed values are guaranteed to be statistically consistent with the observed features.

Traditional regression models fail when the relationship between variables is "one-to-many" (i.e., a single observed input could plausibly correspond to multiple distinct outcomes). In Reconstruction of Sequential Data with Probabilistic Models and Continuity Constraints, Carreira-Perpiñán addressed this by using joint probability modeling combined with global continuity constraints to navigate the multiple modes of a conditional distribution. To handle high-dimensional property constraints where only a partial set of target attributes is provided, Ortega Ochoa et al. utilized GMMs in MolMiner: Towards Controllable, 3D-Aware, Fragment-Based Molecular Design to sample realistic values for unspecified parameters, ensuring a complete and physically plausible input vector for molecular design.

To avoid the limitations of fixed-order autoregressive models, recent deep generative frameworks focus on arbitrary conditioning—the ability to condition on any arbitrary subset of features to predict the rest. Li et al. developed ACFlow: Flow Models for Arbitrary Conditional Likelihoods, a normalizing flow architecture designed specifically for arbitrary conditional likelihoods, providing a unified framework for high-dimensional data imputation.

More recently, order-agnostic architectures and diffusion models have pushed the boundaries of this capability. Zhang et al. proposed TabNAT: A Continuous-Discrete Joint Generative Framework for Tabular Data, which nests conditional diffusion processes within a bi-directional masked Transformer, enabling a single model to perform both full-table synthesis and conditional missing data imputation without task-specific retraining. For sequential and multimodal data, diffusion-based recovery frameworks like those proposed by Wang et al. in Incomplete Multimodality-Diffused Emotion Recognition and Jing et al. in Cross-Domain Conditional Diffusion Models for Time Series Imputation demonstrate remarkable stability, maintaining high imputation accuracy even when up to half of the time-series data or entire modalities are missing.

Learning the Joint Distribution from Incomplete Data

An obvious chicken-and-egg problem arises: how do we train a joint generative model if the training data itself is full of missing values?

The key insight is that we can optimize the model parameters by maximizing the marginal likelihood of only the observed data points during training. Nazábal et al. demonstrated this with the Heterogeneous-Incomplete VAE in Handling Incomplete Heterogeneous Data using VAEs, which computes the variational lower bound loss strictly over observed features. By avoiding ad-hoc preprocessing during training, the model preserves critical statistical relationships that would otherwise be biased.

Traditional Pipeline:
[Incomplete Data] ───> [Ad-hoc Imputation (Mean/Zero)] ───> [Train Discriminative Model] (Biased Correlations)

Joint Generative Pipeline:
[Incomplete Data] ───> [Compute Loss on Observed Only] ───> [Learn True Joint Distribution] (Preserved Geometry)

Similarly, Courville et al. accelerated the EM training of GMMs on incomplete datasets in Efficient EM Training of Gaussian Mixtures with Missing Data by organizing missing data patterns into a Minimum Spanning Tree, achieving massive speedups while maintaining the integrity of conditional mean imputations. For modern score-based and diffusion models, which typically require complete data to compute gradients, Givens et al. introduced Marginal Importance Weighting and Marginal Variational Score Matching in Score Matching with Missing Data. These frameworks allow score matching objectives to integrate agnostically with missing-data mechanisms, enabling robust density estimation across arbitrary missingness patterns.

Instead of directly imputing missing values for individual records, Endres et al. focused on estimating the global joint probability distribution across categorical data files using log-linear Markov networks in Utilizing log-linear Markov networks to integrate categorical data files. For high-dimensional datasets, Saad showed in A Probabilistic Programming Approach To Probabilistic Data Analysis that composite generative models outperform traditional discriminative models by naturally handling incomplete inputs without manual cleaning routines. To handle missing modalities during training, Ma et al. used a generalized softmax function as a mathematical bridge to derive conditional distributions directly from a joint distribution model in Maximum Likelihood Estimation for Multimodal Learning with Missing Modality.

Shared Latent Spaces and Data Fusion

When dealing with complex, heterogeneous data sources—such as multi-view datasets or electronic health records—the joint distribution can be modeled effectively by linking observed variables through a shared latent space.

Instead of directly mapping one view to another, the framework treats missing views as unobserved latent variables. Nguyen developed a hierarchical Bayesian framework in Labeling Complicated Objects: Multi-View Multi-Instance Multi-Label Learning (further explored by Chan et al. in Reconsidering Mutual Information Based Feature Selection: A Statistical Significance View) that leverages learned label correlations and instance distributions to maintain classification performance even when primary data views are entirely absent. Lian et al. showed in Integrating Features and Similarities: Flexible Models for Heterogeneous Multiview Data that linking multiple data views through a shared latent representation avoids the distortions introduced by filling missing values with arbitrary constants.

This latent-driven inference is highly powerful in structured domains:

By shifting the perspective from "fixing missing data" to "modeling the full joint distribution," generative approaches provide a mathematically rigorous and highly flexible solution to incomplete data. Whether through marginalization to bypass missing features or conditional sampling to reconstruct them, joint generative modeling ensures that downstream tasks remain robust, accurate, and statistically sound.

Key Foundational Work

Reconstruction of Sequential Data with Probabilistic Models and Continuity Constraints (1999) — contributed 1 concept elements to this direction.
A Mixture Model System for Medical and Machine Diagnosis (1994) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_variable_imputation_generative_da_06ce4855,
  title = {latent variable imputation (generative data imputation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-variable-imputation-generative-data-imputation/rdr_06ce48554ceb4103896a43efc7d3d5cf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_variable_imputation_generative_da_06ce4855_v1,
  title = {latent variable imputation (generative data imputation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-variable-imputation-generative-data-imputation/rdr_06ce48554ceb4103896a43efc7d3d5cf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake