Expectation Maximization for Learning from Incomplete Data

56 concepts50 papers1 proposal

Real-world datasets are rarely complete. Whether dealing with missing clinical metrics in electronic health records, occluded landmarks in computer vision, or sparse user interactions in collaborative filtering, unobserved values present a fundamental challenge.

When faced with missing data, standard practices often resort to two extremes: discarding incomplete samples entirely (listwise deletion) or filling them in using heuristics like mean imputation or linear interpolation. Discarding data wastes valuable observed features, while heuristic imputation distorts the underlying data distribution, introducing artificial certainties and breaking down downstream statistical models.

To avoid these pitfalls, a powerful framework treats missing data points not as errors to be patched over, but as latent variables—unobserved parameters inherent to the probabilistic model. By framing missing data this way, we can deploy the Expectation-Maximization (EM) algorithm to co-optimize both the missing values and the global model parameters simultaneously.


The Core Mechanism: The EM Feedback Loop

The intuition behind using EM for missing data lies in a cooperative feedback loop between local data completion and global parameter estimation. If we knew the true global parameters of our data distribution (such as the true mean and covariance), filling in the missing values would be a straightforward conditional inference problem. Conversely, if we knew the true values of the missing data, calculating the global parameters would be trivial. Because we know neither, the EM algorithm alternates between estimating one and updating the other.

       ┌────────────────────────────────────────────────────────┐
       │                                                        │
       ▼                                                        │
┌──────────────┐  Conditional Inference  ┌────────────────────┐ │
│ Current Model│ ──────────────────────> │ Expected Statistics│ │
│  Parameters  │ <────────────────────── │  of Missing Data   │ │
└──────────────┘    Parameter Update     └────────────────────┘ │
       │                                                        │
       └────────────────────────────────────────────────────────┘
  1. The Expectation (E) Step: The current iteration of the model parameters is used to compute the conditional expectation of the missing data given the observed data. We are asking: "Given what we currently know about the global data distribution, what are the most likely values or statistical properties for these specific missing entries?"
  2. The Maximization (M) Step: The model updates its global parameters (such as means, covariances, or neural network weights) to maximize the likelihood of the completed dataset.

This creates a powerful optimization trajectory. As the global parameters become more accurate, the imputation of the missing values improves. Conversely, as the missing values are refined, the parameter estimates become more robust. As demonstrated in early foundational work by Ghahramani et al., treating supervised learning as a joint density estimation problem within an EM framework allows models to seamlessly "fill in" missing features without discarding incomplete records.


Bypassing Reconstruction via Direct Estimation

A common misconception is that we must explicitly reconstruct every single missing raw value to learn from incomplete data. In high-dimensional settings, trying to reconstruct raw values can introduce unwanted artifacts and computational bottlenecks. Instead, the EM framework allows for the direct estimation of global statistics—like the covariance matrix—by computing the conditional expectations of sufficient statistics rather than individual data points.

For instance, Sam T. Roweis in EM Algorithms for PCA and SPCA demonstrated that principal components can be estimated directly from incomplete datasets by treating missing coordinates as latent variables, completely bypassing ad-hoc interpolation or data deletion.

This direct estimation strategy proves highly effective across diverse domains:

  • Signal Processing: In brain-computer interface applications, Hippert-Ferrer et al. showed that directly estimating covariance matrices via EM from incomplete EEG signals outperforms traditional signal imputation or masking, avoiding the errors introduced by trying to reconstruct raw high-dimensional time series.
  • Matrix Completion: Léger et al. leveraged a Maximum A Posteriori EM (MAP-EM) algorithm to efficiently complete matrices using a single Gaussian model, updating global mean and covariance parameters based on the conditional expectations of missing entries.
  • Covariance Modeling: Hippert-Ferrer et al. applied the EM framework to estimate low-rank covariance matrices from datasets with arbitrary, non-monotone missingness patterns, recovering the underlying data structure directly from sufficient statistics.

Handling Complex and Heterogeneous Data

The beauty of treating missing data as latent variables is its versatility across diverse data types and domains. It allows models to remain statistically grounded even when the data exhibits complex structural dependencies or severe observation bias.

Clinical and Survival Analysis

In clinical settings, data is notoriously sparse and missingness is rarely random. Ahmed H. Shahin utilized probabilistic latent variable models via EM to preserve statistical relationships between observed and unobserved clinical features for survival analysis, offering a principled alternative to simple mean imputation. To handle non-random observation bias in electronic health records, Dong et al. designed an EM training procedure that alternates between estimating the posterior distribution of complete data and updating model parameters to maximize observed data likelihood.

Structural and Spatial Constraints

In structural and spatial domains, single-author works like Max Welling demonstrated that missing landmarks or occlusions can be seamlessly handled by treating unobserved coordinates as hidden variables within an EM framework to infer shape distributions. Similarly, in clinical clustering, Nino Stocchetti showed that treating incomplete observations as latent variables during the clustering process avoids the biases introduced by traditional data imputation methods.


Scaling to Modern Deep Learning and Distributed Systems

While the EM framework for missing data originated in classical statistics, it has evolved to empower modern deep generative architectures and distributed computing paradigms.

Deep Generative Models

Traditional diffusion models require complete training data and struggle with conditional inference when data is missing. To overcome this, Zhang et al. introduced DiffPuter, a framework that embeds a diffusion model directly inside an EM loop. The process alternates between optimizing the diffusion denoiser on the current completed dataset (M-step) and refining the missing entries using the updated model (E-step).

┌─────────────────────────────────────────────────────────────────┐
│                     The Deep EM Loop (DiffPuter)                │
│                                                                 │
│  ┌──────────────────────┐               ┌────────────────────┐  │
│  │  Diffusion Denoiser  │ ────────────> │  Refined Missing   │  │
│  │  (Optimized on current│               │  Entries (E-Step)  │  │
│  │   completed data)    │ <──────────── │                    │  │
│  └──────────────────────┘               └────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Time-Series Copulas

For multivariate time series, Su et al. developed a Temporal Gaussian Copula that maps irregular clinical records into a latent Gaussian space, using EM to alternate between predicting latent values and updating global covariance parameters. This captures both temporal and cross-variable relationships simultaneously.

Distributed and Federated Learning

When data is siloed across multiple remote sites, missing data handling becomes even more challenging due to privacy constraints. Chang et al. framed distributed learning itself as a missing data problem, where a central node uses EM to impute unobserved sufficient statistics from remote sites without transferring raw records. Similarly, Dieuleveut et al. extended this to federated learning, enabling the collaborative estimation of global population parameters and missing values from distributed, incomplete datasets while mitigating data heterogeneity.

Key Foundational Work

Integrating Probabilistic Rules into Neural Networks: A Stochastic EM Learning Algorithm (1991) — contributed 1 concept elements to this direction.
Improvements in the Stochastic Segment Model for Phoneme Recognition (1989) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_variable_imputation_expectation_m_6de75e70,
  title = {latent variable imputation (expectation-maximization for missing data)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-variable-imputation-expectation-maximization-for-missing-data/rdr_6de75e70fba5419d96f9db5b928aef5b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_variable_imputation_expectation_m_6de75e70_v1,
  title = {latent variable imputation (expectation-maximization for missing data)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-variable-imputation-expectation-maximization-for-missing-data/rdr_6de75e70fba5419d96f9db5b928aef5b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake