Solving Inverse Problems with Score-Based Generative Priors learned from Noisy Data

Asad Aali, Marius Arvinte, Sidharth Kumar, Jonathan I. Tamir

Introduction

Score-based generative models, particularly diffusion models, have emerged as a powerful paradigm for solving inverse problems—the task of recovering a high-fidelity signal from corrupted or incomplete measurements. These models typically function by learning the "score" (the gradient of the log-density) of a data distribution, which serves as a prior during the reconstruction process. However, a significant bottleneck in their application is the "clean data hunger": standard training techniques like Denoising Score Matching (DSM) require massive datasets of noise-free, fully-sampled examples.

In many practical domains, such as medical imaging (MRI) or wireless communications, clean data is either impossible to obtain or prohibitively expensive. In these fields, signals are inherently captured through noisy hardware or sensors. If a generative model is trained naively on this "dirty" data, it often learns to generate the noise itself rather than the underlying signal. This paper introduces SURE-Score, a framework designed to learn high-quality generative priors directly from noisy training samples by leveraging statistical estimators.

Methodology

The core innovation of SURE-Score lies in the elegant integration of three mathematical concepts: Stein’s Unbiased Risk Estimate (SURE), Denoising Score Matching (DSM), and Tweedie’s Formula.

The SURE-Score Framework

  1. Stein’s Unbiased Risk Estimate (SURE): SURE provides a way to estimate the Mean Square Error (MSE) of a denoiser without requiring access to the ground-truth clean signal. It relies solely on the noisy observation and the known noise variance.
  2. Denoising Score Matching (DSM): This is the standard objective for training diffusion models, where the model learns to reverse a perturbation process to estimate the score of the clean data distribution.
  3. Tweedie’s Formula: This identity serves as the bridge, relating the Minimum Mean Square Error (MMSE) denoiser of a Gaussian-perturbed signal directly to the score function of that distribution.

By combining these, the authors formulate a joint loss function. Instead of training a separate denoiser and score model, they use a single deep neural network (based on the RefineNet/NCSNv2 architecture) to represent the score function. By applying Tweedie’s formula, the network is treated as a denoiser within the SURE objective. This ensures the model is self-consistent: it learns to denoise the training data while simultaneously capturing the score function of the underlying clean distribution.

Inference via Annealed Langevin Dynamics

Once the score model is trained, the researchers employ Annealed Langevin Dynamics for inference. This stochastic process starts with pure noise and iteratively "nudges" the estimate toward the learned distribution while enforcing consistency with the specific observed measurements of the inverse problem.

Experimental Evaluation

The researchers evaluated SURE-Score across two distinct and challenging domains to demonstrate its versatility.

Wireless MIMO Channels

In wireless communications, estimating the Multiple-Input Multiple-Output (MIMO) channel is critical. The authors used 10,000 simulated Clustered Delay Line (CDL-C) channel models. Even when the training data was extremely noisy (0 dB SNR, where noise power equals signal power), SURE-Score achieved reconstruction performance within 8 dB of a model trained on perfectly clean data. In contrast, naive training on the same noisy data resulted in significantly degraded performance.

Medical Imaging (MRI)

Using 10,000 T2-weighted brain scans from the fastMRI dataset, the authors simulated a "noisy-only" training environment. For a 5× accelerated MRI reconstruction task, SURE-Score trained on 10 dB noisy data achieved a Normalized Root Mean Square Error (NRMSE) of 0.19. This is remarkably close to the 0.16 NRMSE achieved by a "gold standard" supervised model trained on clean data. Naive training on noisy data failed significantly, yielding an error of 0.71 due to the amplification of training-set noise in the final reconstruction.

Key Insights and Discussion

  • Robustness to Low SNR: The framework's ability to learn useful priors at 0 dB SNR suggests it is highly robust to the quality of available training data.
  • Self-Supervised Consistency: The method prevents overfitting to specific noise realizations by enforcing a consistency objective. It ensures that the gradients remain stable even when noise is removed and re-added at the training level.
  • Qualitative Superiority: Visual analysis of the MRI results indicates that SURE-Score produces sharp, clear images, whereas models trained naively on noisy data suffer from persistent "speckle" artifacts.

Limitations and Future Work

While promising, the current implementation of SURE-Score has some constraints. It primarily assumes independent and identically distributed (i.i.d.) Gaussian noise. Although theoretically extendable to other exponential family distributions (like Poisson noise), this has yet to be empirically validated in this specific framework. Additionally, the SURE loss requires calculating the divergence of the network output, which necessitates a Monte Carlo approximation and an extra forward pass during training, increasing computational overhead. Finally, balancing the SURE and DSM loss components ($\lambda$) remains a hyperparameter challenge in the absence of a clean validation set.

Conclusion

SURE-Score represents a significant advancement in making generative models accessible for scientific and medical applications. By removing the requirement for clean training data, it opens the door for high-fidelity signal reconstruction in fields where "perfect" data is a luxury, effectively turning noisy observations into a powerful resource for learning.

Summary updated

BibTeX
@inproceedings{aali2023solving,
  author = {Asad Aali and Marius Arvinte and Sidharth Kumar and Jonathan I. Tamir},
  title = {{Solving Inverse Problems with Score-Based Generative Priors learned from Noisy Data}},
  booktitle = {ACSSC},
  year = {2023},
  doi = {10.1109/ieeeconf59524.2023.10477042},
}
Figure