Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Images

Kwanyoung Kim, Jong Chul Ye

Deep learning has revolutionized image denoising, but most state-of-the-art models are "supervised"—they require pairs of noisy and perfectly clean images for training. In fields like medical imaging or astronomy, clean ground-truth images are often impossible to obtain.

Existing "self-supervised" alternatives that train only on noisy data have significant drawbacks. For instance, Noise2Noise requires multiple noisy versions of the exact same image. Methods like Noise2Void and Noise2Self work on single images but can struggle with certain noise types or lose fine details. Meanwhile, SURE-based methods are mathematically elegant but highly sensitive to hyperparameters and must be completely retrained if the noise level or model changes.

Noise2Score is a novel self-supervised framework that unifies these disparate methods under a single Bayesian umbrella. By leveraging Tweedie’s formula, the authors demonstrate that denoising any exponential family noise (e.g., Gaussian, Poisson, Gamma) can be reduced to a single task: estimating the "score function" (the gradient of the log-likelihood) of the noisy data.

The Noise2Score Methodology

The core innovation of this paper is the application of Tweedie’s formula, a classical result from Bayesian statistics. It provides an explicit way to calculate the posterior mean (the "best guess" for the clean image) using only the noisy measurement and the score function of the noisy data distribution.

The Noise2Score pipeline breaks the denoising process into two distinct steps:

  1. Score Function Estimation: The authors use an Amortized Residual Denoising Autoencoder (AR-DAE) to estimate the score function from noisy images. Crucially, this training step is "universal"—the network learns the structure of the noise in the data without needing to know the specific noise model (Gaussian vs. Poisson) beforehand.
  2. Tweedie’s Post-processing: Once the score is estimated, the final denoised image is obtained by plugging that score into a specific version of Tweedie’s formula tailored to the noise type.
Overall reconstruction flow comparing standard methods (a) with the proposed Noise2Score (b).

As shown in the comparison above, while standard methods try to map directly from noisy to clean images, Noise2Score first extracts the statistical "score" of the noise before applying a mathematical formula to retrieve the clean image.

Key Insights

  • The Universal Denoiser: Because the score function estimation is independent of specific noise parameters, you do not need to retrain the neural network if the noise level changes. You simply adjust the parameters in the Tweedie formula during inference.
  • Bridging the Gap: The paper mathematically demonstrates that the loss functions used in DAEs and SURE are essentially different ways of trying to find this same "score function." This provides a unified theoretical foundation for the field.
  • Blind Denoising: Noise2Score excels at "blind" setups where the noise level (like the standard deviation $\sigma$ of Gaussian noise) is unknown. By using an image quality metric like Total Variation (TV) during the post-processing step, the optimal noise parameters can be found "on-the-fly" without retraining the model.

Experimental Results

The authors evaluated their method on standard computer vision benchmarks, including BSD68 and Set12 for grayscale, and CBSD68 and Kodak for color images. The models were trained using DIV2K and CBSD400 datasets with synthetic Gaussian, Poisson, and Gamma noise.

Noise2Score consistently outperformed other single-image self-supervised methods across all tested noise types. In Gaussian denoising ($\sigma=25$), Noise2Score achieved a PSNR of 29.12 dB on BSD68, significantly higher than Noise2Void (26.77 dB) and SURE (28.78 dB). Furthermore, the method retains significantly more texture and sharp edges compared to competitors, which often produce over-smoothed results. The performance gains were even more pronounced for non-Gaussian noise (Poisson and Gamma), where traditional methods like SURE are harder to adapt.

Limitations and Future Directions

While highly effective, the current derivation relies on the noise following an exponential family distribution. While this covers the vast majority of common noise types, it might not directly apply to more complex, structured "real-world" noise. Additionally, while the model doesn't need to know the noise parameters during training, the user still needs to know the type of noise (e.g., "this is Poisson noise") to choose the correct Tweedie formula for the final step.

Future work may extend this framework to handle non-exponential noise and explore its application in specific domains like low-dose CT scans or microscopy, where clean data is notoriously scarce.

Summary updated

Abstract

Recently, there has been extensive research interest in training deep networks to denoise images without clean reference.However, the representative approaches such as Noise2Noise, Noise2Void, Stein's unbiased risk estimator (SURE), etc. seem to differ from one another and it is difficult to find the coherent mathematical structure. To address this, here we present a novel approach, called Noise2Score, which reveals a missing link in order to unite these seemingly different approaches.Specifically, we show that image denoising problems without clean images can be addressed by finding the mode of the posterior distribution and that the Tweedie's formula offers an explicit solution through the score function (i.e. the gradient of loglikelihood). Our method then uses the recent finding that the score function can be stably estimated from the noisy images using the amortized residual denoising autoencoder, the method of which is closely related to Noise2Noise or Nose2Void. Our Noise2Score approach is so universal that the same network training can be used to remove noises from images that are corrupted by any exponential family distributions and noise parameters. Using extensive experiments with Gaussian, Poisson, and Gamma noises, we show that Noise2Score significantly outperforms the state-of-the-art self-supervised denoising methods in the benchmark data set such as (C)BSD68, Set12, and Kodak, etc.
BibTeX
@inproceedings{kim2021noise,
  author = {Kwanyoung Kim and Jong Chul Ye},
  title = {{Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Images}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=uQCJDN9Cuvx},
}
Figure