Score-based Diffusion Models for Bayesian Image Reconstruction

Michael T. McCann, Hyungjin Chung, Jong Chul Ye, Marc Klasky

Abstract

This paper explores the use of score-based diffusion models for Bayesian image reconstruction. Diffusion models are an efficient tool for generative modeling and can be adapted for solving inverse problems. We present a simple and flexible algorithm for training a diffusion model at a single noise level and using it for maximum a posteriori (MAP) reconstruction, minimum mean square error (MMSE) reconstruction, and posterior sampling. Experiments on linear and nonlinear reconstruction problems highlight the strengths and limitations of this approach.

Introduction

Image reconstruction—recovering an image $x$ from noisy or incomplete measurements $y$—is a fundamental challenge in fields ranging from medical imaging to astronomy. Traditionally, this is solved using Bayesian inference, which requires a "prior" $p(x)$ to model the distribution of typical images.

While modern diffusion models are excellent at learning these priors, they usually involve complex, multi-step "annealing" processes across many noise scales. This complexity makes it difficult to perform exact Bayesian tasks like MMSE estimation or posterior sampling without significant approximations. This work investigates whether a simplified diffusion model, trained at a single noise level, can effectively serve as a prior for various reconstruction tasks.

Methodology

The core of the proposed method relies on Tweedie’s formula, which provides a mathematical bridge between a denoiser and the "score function" (the gradient of the log-density of the data).

  1. Training: A standard Convolutional Neural Network (CNN) is trained to denoise images corrupted by Gaussian noise at a specific variance $\sigma^2$. This denoiser implicitly learns the score of the noisy data distribution.
  2. Score-based MAP: To find the most likely image (MAP), we perform gradient descent on a loss function where the gradient of the prior term is replaced by the output of the trained denoiser.
  3. Posterior Sampling & MMSE: To explore the range of possible solutions, we use Langevin diffusion. This iterative process uses the learned score function and the measurement model to "nudge" a random noise image toward the posterior distribution. By averaging these samples, we obtain the MMSE reconstruction.

This framework is flexible enough to handle both linear problems (e.g., inpainting) and nonlinear problems (e.g., phase retrieval).

Key Insights: The "Goldilocks" Noise Level

A critical finding is that the noise level used during training significantly impacts the quality of the learned prior. If the training noise is too low, the network only learns local pixel correlations and fails to capture global structures. Conversely, a moderate noise level forces the network to learn broader patterns, making it a much more effective prior for reconstruction.

Samples illustrating the effect of different training noise levels on the learned prior.

Key Results and Discussion

The model was evaluated on a synthetic "Gaussian blobs" dataset ($64 \times 64$ pixels). The results highlight a classic Bayesian trade-off between MAP and MMSE estimations.

  • MAP vs. MMSE: MAP reconstructions often appear sharp but can be noisy or stuck in local minima due to the non-convexity of the prior. In contrast, MMSE reconstructions—obtained by averaging posterior samples—are quantitatively more accurate (higher PSNR) but appear blurry, as they represent the mean of all valid solutions.
  • Uncertainty Quantification: By generating multiple posterior samples, the method provides a measure of uncertainty ($\sigma_{\text{MMSE}}$), indicating where the model is less certain about the reconstruction (e.g., at the edges of structures).
Reconstruction results for a nonlinear problem: (a) ground truth $x$, (b) measurements $y$, (c) MAP reconstructions, (d) posterior samples, (e) MMSE estimate, and (f) uncertainty map.

In quantitative tests, the MMSE approach consistently outperformed MAP. For image inpainting, MMSE achieved a PSNR of 9.79 dB compared to ~6 dB for MAP. Similarly, in Fourier magnitude retrieval, MMSE reached 11.40 dB versus ~5 dB for MAP.

Conclusion and Future Work

This work demonstrates that a single-noise diffusion model can serve as a powerful and flexible prior for Bayesian image reconstruction. While simpler than multi-scale approaches, it effectively handles both linear and nonlinear inverse problems. However, limitations remain: the single-noise approximation means the prior is technically a "noisy" version of the true distribution, and Langevin sampling remains computationally expensive. Future research will explore schemes to reduce noise levels during reconstruction without sacrificing the simplicity of the single-scale framework.

Summary updated

Abstract

This paper explores the use of score-based diffusion models for Bayesian image reconstruction. Diffusion models are an efficient tool for generative modeling. Diffusion models can also be used for solving image reconstruction problems. We present a simple and flexible algorithm for training a diffusion model and using it for maximum a posteriori reconstruction, minimum mean square error reconstruction, and posterior sampling. We present experiments on both a linear and a nonlinear reconstruction problem that highlight the strengths and limitations of the approach.
BibTeX
@inproceedings{mccann2023score,
  author = {Michael T. McCann and Hyungjin Chung and Jong Chul Ye and Marc Klasky},
  title = {{Score-based Diffusion Models for Bayesian Image Reconstruction}},
  booktitle = {ICIP},
  year = {2023},
  url = {https://openreview.net/forum?id=r637IMFJSp},
}
Figure