Real-world datasets are rarely complete. Traditional approaches to missing data often rely on ad-hoc preprocessing, such as dropping incomplete rows or filling them in with a single deterministic value like the mean or a regression prediction. The fundamental flaw in these deterministic approaches is artificial certainty: when a downstream model receives a filled-in value, it treats it with the same confidence as a true observation. This underestimation of variance introduces severe downstream bias and distorts statistical significance.
The Bayesian paradigm offers a more principled alternative. Instead of fixing missing data before modeling, it incorporates the missingness directly into the probabilistic framework. Missing values are treated either as dimensions to be integrated out or as latent random variables to be jointly inferred. By doing so, the uncertainty of what might have been observed is preserved and propagated through to the final parameters.
Marginalization: Integrating Out the Unknown
When a data point or an entire modality is missing, we do not necessarily need to guess its exact value to learn from the remaining data. Instead, we can use Bayesian marginalization.
Mathematically, if we have observed data $X_{\text{obs}}$ and missing data $X_{\text{mis}}$, a standard model defines a joint likelihood $P(X_{\text{obs}}, X_{\text{mis}} | \theta)$ given parameters $\theta$. Rather than substituting a proxy value for $X_{\text{mis}}$, marginalization sums or integrates over all possible values that $X_{\text{mis}}$ could conceivably take:
$$P(X_{\text{obs}} | \theta) = \int P(X_{\text{obs}}, X_{\text{mis}} | \theta) dX_{\text{mis}}$$
As Thiesson (1997) demonstrated in the context of recursive exponential models, this process allows for robust probabilistic inference by accounting for every potential way an incomplete observation could be completed, weighting each by its likelihood.
The primary benefit of this approach is that it completely avoids the bias introduced by synthetic data generation. For instance, in multimodal medical diagnostics where entire imaging modalities (such as MRIs or EEGs) might be missing for certain patients, Akbar et al. (2022) utilized Bayesian marginalization to treat missing attributes as variables to be integrated out. This maintained the ability to utilize incomplete datasets without introducing the artifacts of synthetic imputation. Similarly, Fox et al. (2015) showed that a Bayesian nonparametric framework can naturally accommodate missing data within its core inference process, bypassing explicit imputation steps entirely and simplifying the modeling pipeline.
Joint Inference via Latent Variable Imputation
When marginalization is analytically intractable, or when estimating the missing values themselves is a primary goal, the Bayesian framework treats missing entries as latent random variables. Instead of a two-step "impute then analyze" pipeline, imputation and parameter estimation happen simultaneously within a single joint inference loop.
This is typically achieved using Markov Chain Monte Carlo (MCMC) methods like Gibbs sampling. In each iteration, the algorithm performs two alternating steps:
- Imputation Step: Sample the missing data from its predictive distribution, conditioned on the current model parameters and the observed data.
- Parameter Step: Sample the model parameters from their posterior distribution, conditioned on the observed data and the newly sampled missing data.
Because the missing values are resampled at every iteration, the model never anchors to a single incorrect guess. Gu (2008) leveraged this mechanism for the biclustering of gene expression data, demonstrating that Monte Carlo imputation embedded within Gibbs sampling allows for the seamless estimation of missing values alongside cluster assignments.
This unified view transforms missing data from a data-cleaning nuisance into an intrinsic part of the model. Tung et al. (2022) formalized this concept by designing a domain-specific language, BayesLDM, which automatically treats missing values as unknown random variables to be inferred alongside parameters, ensuring statistical consistency without separate preprocessing steps. Whether tracking hidden infection states in epidemiological networks (Dong et al. (2012)) or sampling from predictive distributions to maintain the integrity of causal forecasting systems (Chu et al. (2024)), joint latent variable imputation ensures that downstream predictions account for the uncertainty inherent in incomplete observations.
Correcting Heuristics with Importance Sampling
While joint MCMC estimation is statistically rigorous, sampling high-dimensional missing data at every iteration can be computationally expensive. Can we exploit fast, heuristic imputation methods without sacrificing Bayesian rigor?
Riggelsen et al. (2005) introduced a clever solution to this trade-off using Importance Sampling. Instead of drawing samples from the true, complex posterior distribution of the missing data, one can generate candidate completions using a computationally inexpensive proposal distribution—such as a cheap heuristic model.
To correct for the fact that this heuristic distribution is biased, each completed dataset is assigned an importance weight proportional to the ratio of the true distribution to the heuristic distribution:
$$W = \frac{P(\text{True Distribution})}{P(\text{Heuristic Distribution})}$$
These weights mathematically correct the bias of the cheap imputation model. By weighting the approximate imputations during structure learning, the true posterior distribution can be recovered without requiring a perfect generative model for the missing values. This allows practitioners to exploit the speed of fast heuristics while maintaining exact Bayesian guarantees.
Exploiting Spatiotemporal and Causal Structures
A key reason this probabilistic framing works so well is its ability to exploit structural dependencies across time, space, or causal graphs. When data is Missing at Random (MAR) or Missing Completely at Random (MCAR), the missingness mechanism itself does not need to be explicitly modeled (Iseki et al. (2019)). Instead, the model relies on surrounding context to fill in the blanks.
For longitudinal and time-series data, frameworks like Bayesian Gaussian Processes naturally bridge gaps over missing intervals by leveraging temporal correlation (Filippone et al. (2019)). In spatial settings, such as camera networks experiencing occlusions or sensor failures, the global model parameters can be robustly estimated by adjusting local summary statistics to include only observed features, treating the occluded views as hidden variables within a shared probabilistic framework (Yoon et al. (2012)).
Ultimately, by viewing missing data not as holes to be plugged, but as unobserved nodes in a broader probabilistic graph, Bayesian inference provides a unified, mathematically rigorous strategy for learning from the messy reality of incomplete data.
Key Foundational Work
Learning Bayesian Network Models from Incomplete Data using Importance Sampling (2005) — contributed 2 concept elements to this direction.
Score and Information for Recursive Exponential Models with Incomplete Data (1997) — contributed 1 concept elements to this direction.