An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations

Weimin Bai, Yifei Wang, Wenzheng Chen, He Sun

Abstract

Diffusion models excel in solving imaging inverse problems due to their ability to model complex image priors. However, their reliance on large, clean datasets for training limits their practical use where clean data is scarce. In this paper, we propose EMDiffusion, an expectation-maximization (EM) approach to train diffusion models from corrupted observations. Our method alternates between reconstructing clean images from corrupted data using a known diffusion model (E-step) and refining diffusion model weights based on these reconstructions (M-step). This iterative process leads the learned diffusion model to gradually converge to the true clean data distribution. We validate our method through extensive experiments on diverse computational imaging tasks, including random inpainting, denoising, and deblurring, achieving new state-of-the-art performance.

Introduction

Diffusion Models (DMs) have emerged as the gold standard for solving inverse problems—tasks such as denoising, inpainting, and deblurring. By learning a sophisticated mathematical prior of natural images, these models can reconstruct high-fidelity details from degraded inputs. However, a significant bottleneck remains: training these models typically requires massive datasets of pristine, "clean" images.

In many scientific and medical domains, such as CT imaging, satellite surveillance, or microscopy, clean data is nearly impossible to obtain. Researchers are often left with a "chicken-and-egg" dilemma: to reconstruct a clean image, one needs a DM trained on clean data; but to get clean data for training, one needs a way to reconstruct it from corrupted observations. Previous attempts to train DMs directly on corrupted data often resulted in "mode collapse" or overly smoothed, blurry outputs. EMDiffusion addresses this gap by providing a framework to learn the true data distribution directly from corrupted observations.

The EMDiffusion Framework

EMDiffusion treats the underlying clean images as latent variables within an Expectation-Maximization (EM) framework. The algorithm iteratively improves the model through two alternating steps:

1. The E-step (Expectation)

In this step, the algorithm takes the corrupted observations and uses the current iteration of the diffusion model to estimate the corresponding clean images. This reconstruction is performed via Diffusion Posterior Sampling (DPS), which leverages the model's current understanding of image priors to "guess" the missing or corrupted information.

2. The M-step (Maximization)

The algorithm then treats these reconstructed images as a new training set. The weights of the diffusion model are retrained and refined based on these reconstructions, effectively updating the model's prior to be more accurate in the next iteration.

Adaptive Likelihood Scaling

A critical innovation in this process is the use of Adaptive Likelihood Scaling ($\\lambda$). At the start of training, the diffusion model is relatively inaccurate. If the E-step relies too heavily on a weak prior, the reconstructions will be poor, leading to a cycle of stagnation. To prevent this, the authors introduce $\\lambda$ to weight the "data likelihood"—how well the reconstruction matches the actual corrupted observation. Early in the process, $\\lambda$ is kept high to force the model to stay true to the observed data. As the model matures, $\\lambda$ is tuned down, allowing the learned prior to take more control and generate sharper, more realistic details.

Experimental Setup

The researchers evaluated EMDiffusion on two standard benchmarks: CIFAR-10 (32x32 resolution) and CelebA (64x64 resolution). The model was tested against three primary types of corruption:

  • Random Inpainting: 60% of pixels were randomly masked.
  • Gaussian Denoising: Heavy noise was added to the images.
  • Gaussian Deblurring: Images were blurred using a $9\times9$ kernel.

To initiate the process, the model uses a "warm start" by training on a tiny seed of just 50 clean images. Remarkably, these 50 images can even come from a different dataset (e.g., using face images to initialize a model for general objects), as the basic statistics of natural images—like edges and textures—are sufficiently similar to begin the EM cycle.

Key Results and Insights

EMDiffusion significantly outperformed existing self-supervised baselines, such as AmbientDiffusion and SURE-Score.

  • Performance Metrics: On the CIFAR-10 inpainting task, EMDiffusion achieved a PSNR of 24.70, nearly matching the performance of a model trained on 100% clean data (25.44) and far exceeding AmbientDiffusion (20.57).
  • Generative Quality: The FID score, which measures the realism of generated images, was 21.08 for EMDiffusion. This is a massive improvement over SURE-Score (220.01), demonstrating that EMDiffusion learns the true distribution rather than simply memorizing noise.
  • Self-Correction: The iterative nature of the EM framework allows the model to "pull itself up by its bootstraps." Each M-step provides a better prior, which in turn makes the next E-step's reconstructions cleaner, creating a virtuous cycle of improvement.

Limitations and Future Work

While EMDiffusion represents a major step forward, it has some limitations. The requirement for 50 clean images, though small, may still be a hurdle in extreme scenarios; future work may explore using handcrafted denoisers to create this initial seed. Additionally, the iterative nature of the algorithm makes it more computationally expensive than standard training. Finally, the authors noted that the DPS sampling method can occasionally introduce sharp artifacts, suggesting that more principled posterior sampling methods could further refine the results.

Conclusion

EMDiffusion provides a robust path for training high-quality diffusion models in data-scarce environments. By successfully bridging the gap between corrupted observations and clean generative priors, it opens new possibilities for high-fidelity imaging in medical and scientific fields where pristine data is a rarity.

Summary updated

Abstract

Diffusion models excel in solving imaging inverse problems due to their ability to model complex image priors. However, their reliance on large, clean datasets for training limits their practical use where clean data is scarce. In this paper, we propose EMDiffusion, an expectation-maximization (EM) approach to train diffusion models from corrupted observations. Our method alternates between reconstructing clean images from corrupted data using a known diffusion model (E-step) and refining diffusion model weights based on these reconstructions (M-step). This iterative process leads the learned diffusion model to gradually converge to the true clean data distribution. We validate our method through extensive experiments on diverse computational imaging tasks, including random inpainting, denoising, and deblurring, achieving new state-of-the-art performance.
BibTeX
@inproceedings{bai2024expectation,
  author = {Weimin Bai and Yifei Wang and Wenzheng Chen and He Sun},
  title = {{An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=jURBh4V9N4},
}
Figure