Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Images

Kwanyoung Kim, Jong Chul Ye

Abstract

Recent advancements in deep learning have enabled image denoising without clean reference data, yet existing methods like Noise2Noise, Noise2Void, and Stein’s Unbiased Risk Estimator (SURE) lack a unified mathematical framework. We present Noise2Score, a novel approach that bridges these disparate techniques using Tweedie’s formula. By estimating the score function (the gradient of the log-likelihood) from noisy images via an amortized residual denoising autoencoder, Noise2Score provides a universal solution for any noise distribution within the exponential family. Our experiments demonstrate that Noise2Score significantly outperforms state-of-the-art self-supervised methods on benchmark datasets, achieving performance levels comparable to supervised learning.

Introduction

In many high-stakes imaging domains, such as medical diagnostics, satellite surveillance, and microscopy, obtaining clean "ground truth" images for training denoising models is often impossible. This has led to the rise of self-supervised denoising, where models learn directly from noisy observations. However, the field has been characterized by a collection of specialized heuristics: Noise2Noise requires multiple noisy realizations of the same scene, while Noise2Void employs "blind-spot" architectures to avoid learning identity mappings.

Noise2Score addresses this fragmentation by identifying a "missing link" in the mathematical structure of self-supervised learning. By leveraging Tweedie’s formula, the authors provide a principled Bayesian framework that treats denoising as a posterior mean estimation problem, unified by the concept of the "score function."

Theoretical Foundation: Tweedie’s Formula

The core innovation of Noise2Score is the application of Tweedie’s formula, a classical result in Bayesian statistics. This formula allows for the calculation of the posterior mean—the optimal estimate of a clean image—using only the noisy observation and the score function of the noisy data distribution.

Crucially, the score function (the gradient of the log-likelihood) captures the underlying structure of the image data. The authors demonstrate that the training objectives of popular methods like Noise2Void are essentially approximations of this score function. By making this relationship explicit, Noise2Score moves self-supervised denoising from a set of architectural tricks to a rigorous statistical inference problem.

The Noise2Score Pipeline

The Noise2Score method decouples the learning of image structures from the specific characteristics of the noise. The process follows a two-step pipeline:

  1. Universal Score Estimation (Training): An Amortized Residual Denoising Autoencoder (AR-DAE) is trained on noisy images. The network learns to estimate the score function by removing small amounts of synthetic noise added during training. This phase is "universal" because the network learns the prior distribution of the images, which is independent of whether the final target noise is Gaussian, Poisson, or Gamma.
  2. Noise-Specific Inference: During the inference phase, the pre-trained score function is plugged into the specific version of Tweedie’s formula that corresponds to the noise type at hand. This allows a single trained model to be adapted to various noise models (e.g., Gaussian for electronics, Poisson for photon counting) without retraining.

Experimental Results

The authors evaluated Noise2Score against standard benchmarks (BSD68, Set12, CBSD68, and Kodak) and real-world microscopy data (FMD).

  • Quantitative Performance: Noise2Score consistently outperformed existing self-supervised baselines, including Noise2Void and SURE. In many benchmarks, it achieved a PSNR gain of 0.5 to 3.0 dB, narrowing the gap between self-supervised and fully supervised models.
  • Blind Denoising: The framework allows for "on-the-fly" estimation of unknown noise parameters. By minimizing image quality metrics like Total Variation during inference, the model can adapt to unknown noise levels in seconds, a significant improvement over methods requiring hours of retraining.
  • Real-World Application: On the Fluorescence Microscopy Dataset (FMD), which features complex mixed Poisson-Gaussian noise, the two-step Tweedie approach proved more robust than specialized self-supervised competitors.

Discussion and Limitations

While Noise2Score provides a powerful unified theory, it is currently constrained to noise distributions within the exponential family. While this covers the majority of practical use cases (Gaussian, Poisson, Gamma, Bernoulli), it may not directly apply to highly complex, non-parametric noise found in some raw sensor data. Furthermore, as with any high-performance denoiser, there is a risk of over-smoothing fine textures, which remains a critical consideration for medical applications where subtle details are diagnostically significant.

Conclusion

Noise2Score represents a significant step forward in self-supervised image restoration. By unifying existing methods under the umbrella of Tweedie’s formula and score estimation, it provides a mathematically grounded, flexible, and high-performing framework that eliminates the need for clean reference images across a wide variety of scientific and industrial applications.

Summary updated

Abstract

Recently, there has been extensive research interest in training deep networks to denoise images without clean reference. However, the representative approaches such as Noise2Noise, Noise2Void, Stein's unbiased risk estimator (SURE), etc. seem to differ from one another and it is difficult to find the coherent mathematical structure. To address this, here we present a novel approach, called Noise2Score, which reveals a missing link in order to unite these seemingly different approaches. Specifically, we show that image denoising problems without clean images can be addressed by finding the mode of the posterior distribution and that the Tweedie's formula offers an explicit solution through the score function (i.e. the gradient of log likelihood). Our method then uses the recent finding that the score function can be stably estimated from the noisy images using the amortized residual denoising autoencoder, the method of which is closely related to Noise2Noise or Nose2Void. Our Noise2Score approach is so universal that the same network training can be used to remove noises from images that are corrupted by any exponential family distributions and noise parameters. Using extensive experiments with Gaussian, Poisson, and Gamma noises, we show that Noise2Score significantly outperforms the state-of-the-art self-supervised denoising methods in the benchmark data set such as (C)BSD68, Set12, and Kodak, etc.
BibTeX
@inproceedings{kim2021noise,
  author = {Kwanyoung Kim and Jong Chul Ye},
  title = {{Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Images}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=uQCJDN9Cuvx},
}
Figure