An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations

Weimin Bai, Yifei Wang, Wenzheng Chen, He Sun

TL;DR

EMDiffusion is a new framework that allows researchers to train high-quality diffusion models even when a clean dataset is unavailable. By using an Expectation-Maximization (EM) approach, the method iteratively reconstructs clean images from corrupted observations and refines the model, eventually converging to the true data distribution using as few as 50 clean images for initialization.

What Problem Does This Paper Address?

Diffusion models (DMs) have become the gold standard for generative modeling and solving inverse problems like denoising or inpainting. However, they have a "clean data" problem: to train a DM that can recover a sharp image from a blurry one, you typically need a massive dataset of perfectly sharp images to begin with.

In many scientific and medical fields—such as X-ray CT or cryo-electron microscopy—clean data is a luxury. We often only have access to "corrupted" observations (noisy, blurry, or incomplete). This creates a chicken-and-egg dilemma: you need a clean dataset to train a good prior, but you need a good prior to reconstruct clean data from your observations. Existing workarounds, like AmbientDiffusion, often struggle with noise or produce overly smooth results.

Methods

The authors propose EMDiffusion, which treats the underlying clean images as latent variables in an Expectation-Maximization framework. The process follows a two-step cycle:

  1. The E-step (Expectation): Given the current (initially "vague") diffusion model, the algorithm performs Adaptive Diffusion Posterior Sampling. It tries to reconstruct clean versions of the corrupted observations.
  2. The Innovation: Because the model is weak at the start, standard sampling often fails or "collapses" into a few known examples. The authors introduce a scaling hyperparameter ($\lambda$) to the likelihood term, balancing the model's prior knowledge with the actual evidence in the corrupted observation.
  3. The M-step (Maximization): The algorithm takes these newly reconstructed "cleaner" images and uses them to retrain and refine the diffusion model’s weights via standard score matching.

This cycle repeats. As the model improves, the reconstructions in the E-step become more accurate, which in turn provides better training data for the M-step.

**Figure 1: Overview of EMDiffusion.** The process alternates between E-steps (reconstructing images) and M-steps (updating the model).

Datasets

The researchers validated EMDiffusion on two standard benchmarks:

  • CIFAR-10: 50,000 images at 32x32 resolution.
  • CelebA: 30,000 human face images at 64x64 resolution.

To simulate real-world scarcity, they initialized the process using only 50 clean images—a tiny fraction (0.1%) of the total dataset. They also tested Out-of-Distribution (OOD) initialization, using face images to kickstart a model intended for general objects.

Key Insights and Ideas

  • The Power of $\lambda$: The adaptive scaling factor is the "secret sauce." Early in training, the model's prior is unreliable. By cranking up the importance of the observation (the likelihood) via $\lambda$, the authors prevent the model from ignoring the data and falling into mode collapse.
  • Robust Initialization: Surprisingly, the initial clean data doesn't even have to be from the same distribution. Using 50 faces to initialize a model for CIFAR-10 (cars, birds, etc.) works because natural images share fundamental low-frequency structures like edges and gradients.
  • Self-Correction: The iterative nature allows the model to "pull itself up by its bootstraps." Figure 3 illustrates this beautifully: the first iteration's output is noisy, but by the final iteration, the model produces crisp images that rival those trained on 100% clean data.
**Figure 3: Results on CIFAR-10 inpainting.** Note how the reconstruction quality improves significantly from the 1st iteration to the final iteration.

Key Results

EMDiffusion achieved new state-of-the-art performance for training from corrupted data:

  • Inpainting: On CIFAR-10 with 60% of pixels missing, EMDiffusion reached a PSNR of 24.70, very close to the "upper bound" of 25.44 achieved by a model trained on a fully clean dataset.
  • Denoising & Deblurring: It significantly outperformed baselines like SURE-Score and AmbientDiffusion. For example, in CelebA deblurring, it achieved an FID of 91.89, compared to SURE-Score's 191.96.
  • Efficiency: Despite the iterative training, the authors used a vanilla DDPM architecture with fewer parameters (35.7M) than some baselines (113M), proving the algorithmic approach is more important than raw model size.

Limitations and Future Work

  • Known Forward Models: The current method assumes we know the "corruption process" (e.g., the specific blur kernel or noise level). In the wild, these might be unknown, suggesting a need for "blind" EMDiffusion.
  • E-step Artifacts: The posterior sampling method used (DPS) can sometimes introduce sharp artifacts, which negatively impacts the FID score even if the images look good to the human eye. The authors suggest that designing a more principled posterior sampling method could further boost performance.
  • Computational Cost: Iterative training is naturally more expensive than a single pass, though the authors mitigated this by using subsets of data in early iterations.

Summary updated

Abstract

Diffusion models excel in solving imaging inverse problems due to their ability to model complex image priors. However, their reliance on large, clean datasets for training limits their practical use where clean data is scarce. In this paper, we propose EMDiffusion, an expectation-maximization (EM) approach to train diffusion models from corrupted observations. Our method alternates between reconstructing clean images from corrupted data using a known diffusion model (E-step) and refining diffusion model weights based on these reconstructions (M-step). This iterative process leads the learned diffusion model to gradually converge to a local optimum, that is, to approximate the true clean data distribution. We validate our method through extensive experiments on diverse computational imaging tasks, including random inpainting, denoising, and deblurring, achieving new state-of-the-art performance.
BibTeX
@inproceedings{bai2024expectation,
  author = {Weimin Bai and Yifei Wang and Wenzheng Chen and He Sun},
  title = {{An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=jURBh4V9N4},
}
Figure