Unsupervised Learning with Stein's Unbiased Risk Estimator

Christopher A. Metzler, Ali Mousavi, Reinhard Heckel, Richard G. BaraniukarXiv 2018 · OpenReview

Abstract

Learning from unlabeled and noisy data is one of the grand challenges of machine learning. In this work, we revisit a classical idea: Stein's Unbiased Risk Estimator (SURE). We show that, in the context of image recovery, SURE and its generalizations can be used to train convolutional neural networks (CNNs) for a range of image denoising and recovery problems without any ground truth data. Specifically, our goal is to reconstruct an image $x$ from a noisy linear transformation of the image. We consider scenarios where no additional data is available or where we only have access to other noisy measurements. We demonstrate that SURE can estimate the mean-squared-error (MSE) loss associated with an estimate of $x$, allowing us to train networks for denoising and compressed sensing. Furthermore, we use the SURE framework to explain and improve upon the "Deep Image Prior" phenomenon, where a network fit to a single noisy image can effectively denoise it.

The Ground Truth Bottleneck

Deep learning has revolutionized image recovery tasks like denoising and super-resolution. However, training these models typically requires massive datasets of paired images: a corrupted input and a perfectly clean "ground truth" target. In many real-world domains—such as medical imaging (MRI, CT), microscopy, and astronomy—capturing noise-free ground truth data is either physically impossible or prohibitively expensive. This paper addresses this bottleneck by asking: Can we train state-of-the-art image recovery networks using only noisy, unlabeled measurements?

Methodology: Training with SURE

The core engine of this approach is Stein’s Unbiased Risk Estimator (SURE). SURE is a statistical theorem that provides an unbiased estimate of the MSE of an estimator, even when the true underlying signal is unknown, provided the noise is Gaussian. The SURE formula balances two components:

  1. Data Fidelity: How well the network's output matches the noisy input.
  2. Divergence: How sensitive the network's output is to tiny changes in the input.

Because calculating the exact divergence of a deep neural network is computationally intensive, the authors employ Monte-Carlo SURE (MC-SURE). This method approximates the divergence using a single random noise perturbation, making it feasible to use as a loss function for training CNNs.

Preventing Overfitting

A key insight of this work is how the divergence term in SURE acts as a natural regularizer. In the context of "Deep Image Prior" (DIP)—where a network is optimized on a single noisy image—standard loss functions eventually cause the network to memorize the noise, leading to a degradation in image quality (overfitting).

Comparison of training with standard data fidelity loss (a) versus the proposed SURE loss (b). SURE prevents the NMSE from rising by penalizing network divergence.

As shown in the figure above, when minimizing only data fidelity, the true error (NMSE) begins to rise after approximately 40 iterations. By contrast, the SURE loss tracks the true NMSE closely and prevents the network from fitting the noise, eliminating the need for arbitrary early stopping.

Key Results

The authors applied SURE-based training to three scenarios: untrained single-image denoising, training on noisy datasets, and compressive sensing.

  • Untrained Denoising: A U-Net optimized on a single noisy image using SURE achieved a PSNR of 26.3 dB on the Mandrill image, matching the performance of the classic BM3D algorithm.
  • Noisy Dataset Training: A DnCNN trained exclusively on noisy images from the BSD-500 dataset performed nearly as well as a model trained on clean images (e.g., 29.0 dB vs. 29.4 dB on the Barbara image).
  • Compressive Sensing: The LDAMP network trained with SURE on undersampled measurements achieved performance on par with BM3D-AMP while running significantly faster at test time.
Qualitative comparison of denoising results on the Mandrill image. The untrained U-net using SURE (d) matches the performance of established supervised and traditional methods.

Conclusion

By leveraging Stein's Unbiased Risk Estimator, this research demonstrates that ground truth data is not a strict requirement for training high-performance image recovery networks. The inclusion of a divergence penalty not only allows for unsupervised training but also provides a theoretical foundation for understanding and improving self-supervised techniques like Deep Image Prior. While the method introduces some computational overhead, it opens new doors for deep learning applications in fields where clean data is a luxury.

Summary updated

Abstract

Learning from unlabeled and noisy data is one of the grand challenges of machine learning. As such, it has seen a flurry of research with new ideas proposed continuously. In this work, we revisit a classical idea: Stein's Unbiased Risk Estimator (SURE). We show that, in the context of image recovery, SURE and its generalizations can be used to train convolutional neural networks (CNNs) for a range of image denoising and recovery problems without any ground truth data. Specifically, our goal is to reconstruct an image $x$ from a noisy linear transformation (measurement) of the image. We consider two scenarios: one where no additional data is available and one where we have measurements of other images that are drawn from the same noisy distribution as $x$, but have no access to the clean images. Such is the case, for instance, in the context of medical imaging, microscopy, and astronomy, where noise-less ground truth data is rarely available. We show that in this situation, SURE can be used to estimate the mean-squared-error loss associated with an estimate of $x$. Using this estimate of the loss, we train networks to perform denoising and compressed sensing recovery. In addition, we also use the SURE framework to partially explain and improve upon an intriguing results presented by Ulyanov et al. in "Deep Image Prior": that a network initialized with random weights and fit to a single noisy image can effectively denoise that image. Public implementations of the networks and methods described in this paper can be found at https://github.com/ricedsp/D-AMP_Toolbox.
BibTeX
@misc{metzler2018unsupervised,
  author = {Christopher A. Metzler and Ali Mousavi and Reinhard Heckel and Richard G. Baraniuk},
  title = {{Unsupervised Learning with Stein's Unbiased Risk Estimator}},
  year = {2018},
  eprint = {1805.10531},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.10531},
}
Figure