TL;DR
This paper introduces a novel two-stage approach for image denoising that leverages flow-based generative models to learn a realistic prior from clean images. This allows for effective denoising even when paired noisy and clean images are not available, outperforming existing unsupervised methods and rivaling supervised ones in qualitative results.
What Problem Does This Paper Address?
Image denoising aims to recover a clean image from its noisy counterpart. Traditionally, this has been a supervised learning problem, requiring pairs of noisy and clean images for training. While recent methods like Noise2Void have emerged that can denoise using only noisy images, they don't fully exploit the information present in available clean images. This paper addresses the challenge of image denoising in a setting where clean images are available but not necessarily paired with the noisy images, aiming to utilize the valuable information in clean images for improved denoising performance without full supervision.
Methods
The proposed method employs a two-stage approach:
- Stage 1: Training the Flow Model: A flow-based generative model is trained on clean images. Flow-based models learn an invertible transformation from a high-dimensional data distribution (images) to a simple distribution (Gaussian). This allows them to accurately capture the likelihood function of clean images, effectively learning a realistic prior. The training objective is to minimize the negative log-likelihood of the clean images.
- Stage 2: Training the Denoiser: A denoising network is trained using a loss function that combines two terms:
- A term measuring the squared error between blurred versions of the noisy input and the denoised output. This encourages the denoiser to preserve low-frequency information from the noisy image while adding details to make it cleaner.
- A term derived from the learned prior from Stage 1, which penalizes outputs that are unlikely to be clean images according to the flow model.
This two-stage process allows the denoiser to learn from the prior learned from clean images without requiring direct noisy-clean pairs for its own training.
Datasets
- Training: MS COCO dataset (a subset of 20K images for clean images, and another subset of 20K images with added Gaussian noise for noisy images).
- Validation: A subset of 1K images from MS COCO for hyperparameter tuning.
- Evaluation: BSD68 dataset for quantitative and qualitative comparisons.
Key Insights and Ideas
- Flow-based models as priors: The paper demonstrates that flow-based generative models can serve as powerful, mathematically justifiable priors for image denoising, effectively capturing the distribution of clean images.
- Unsupervised denoising with clean image information: By learning a prior from clean images, the method can improve denoising performance even when direct noisy-clean pairs are unavailable, bridging the gap between fully supervised and purely unsupervised methods.
- Weak supervision for denoising: The second stage utilizes a form of weak supervision by combining a reconstruction loss with the learned prior, allowing the denoiser to learn without explicit clean targets.
- Robustness to noise levels: The method is designed to be agnostic to noise levels, with experiments conducted across a range of standard deviations.
Key Results
- Quantitative: The method achieves competitive PSNR values, performing comparably to Noise2Void and outperforming Deep Image Prior on the BSD68 dataset across various noise levels (σ=15, 25, 35). For σ=25, the method achieved a PSNR of 28.61 dB.
- Qualitative: Visual results show that the proposed method effectively removes noise while preserving fine details and textures, avoiding the blurring or desaturation seen in some other methods. The paper highlights the restoration of fine details in trees and the avoidance of visible noise in the sky as key qualitative successes.
Limitations and Future Work
- Hyperparameter sensitivity: The authors note that the performance of the denoiser is sensitive to the hyperparameter λ, which controls the balance between the reconstruction loss and the prior term. Reducing this dependency on noise level was addressed by using a blurred version of the input, but it remains a consideration.
- Future Work: The authors conjecture that their flow-based prior approach could be effective for other image restoration tasks such as image deblurring and super-resolution in an unsupervised fashion.
Summary updated