Self-supervised Deep Image Restoration via Adaptive Stochastic Gradient Langevin Dynamics

Weixi Wang, Ji Li, Hui Ji

Introduction

Image restoration is a fundamental task in computer vision, encompassing deblurring, denoising, and reconstruction from incomplete measurements. While supervised deep learning models have achieved state-of-the-art performance, their reliance on large datasets of paired "corrupted" and "clean" images limits their applicability in domains where ground truth is unavailable, such as medical MRI or satellite imaging. This paper presents a self-supervised framework that leverages a generative neural network and a novel sampling algorithm to perform high-quality restoration without clean training data.

The Restoration Problem

Traditional unsupervised methods, such as the Deep Image Prior (DIP), utilize the architectural bias of convolutional neural networks (CNNs) as a regularizer. However, DIP is susceptible to overfitting, where the network eventually begins to reconstruct noise if training proceeds too long. Bayesian approaches offer a more robust alternative by estimating the posterior distribution of the image. The authors specifically target the Minimum Mean Squared Error (MMSE) estimator, which represents the average of all images consistent with the observed data.

Methodology: Adaptive Stochastic Gradient Langevin Dynamics (ASGLD)

The core innovation of this work is the Adaptive Stochastic Gradient Langevin Dynamics (ASGLD) algorithm. In a Bayesian setting, the image is represented by the weights of a deep CNN. Instead of point estimation, the goal is to sample from the posterior distribution of these weights.

Standard SGLD uses a constant noise term to explore the solution space. ASGLD improves upon this by making the noise variance adaptive to the current loss. When the reconstruction error is high, the noise is minimized to allow for rapid descent toward the feasible region. As the error decreases, the noise increases, preventing the model from collapsing into a single noisy solution and instead forcing it to sample the distribution of likely images.

PCA visualization of the ASGLD sampling process, showing how samples converge toward the ground truth and the Bayesian average.

Key Insights: The Feasible Set and Self-Regularization

  1. The Feasible Set: For high-dimensional data, the "true" solution typically resides within a narrow set where the reconstruction error matches the expected noise level. ASGLD is designed to trap samples within this feasible set, ensuring consistent and accurate reconstructions.
  2. Self-Regularization through Averaging: By averaging multiple samples generated by ASGLD, the random noise inherent in individual samples cancels out, while the underlying image structure—which is consistent across samples—is preserved. This eliminates the need for the "early stopping" heuristic required by DIP.

Experimental Results

The proposed method was evaluated across various tasks, including natural image reconstruction, medical MRI, and phase retrieval. ASGLD consistently outperformed other unsupervised benchmarks like DIP and Bayesian Neural Networks (BNN) by 0.5 to 1.5 dB in Peak Signal-to-Noise Ratio (PSNR).

Visual result of image restoration using the ASGLD framework, demonstrating high fidelity and noise suppression.

Notably, ASGLD achieved results competitive with supervised models like ISTA-Net, despite the lack of clean training data. Furthermore, the adaptive nature of the algorithm makes it approximately three times faster than traditional SGLD-based approaches.

Conclusion and Future Work

The ASGLD framework provides a powerful, self-supervised solution for inverse problems in imaging. By combining the structural priors of deep networks with an efficient adaptive Bayesian sampling strategy, it overcomes the overfitting limitations of previous unsupervised methods while maintaining computational efficiency. While the method requires per-image optimization and is sensitive to the choice of the noise constant ($c_0$), it represents a significant step toward practical unsupervised image restoration. Future work will explore the application of this framework to more complex ill-posed problems such as blind deconvolution.

Summary updated

Abstract

While supervised deep learning has been a prominent tool for solving many image restoration problems, there is an increasing interest on studying self-supervised or un-supervised methods to address the challenges and costs of collecting truth images. Based on the neuralization of a Bayesian estimator of the problem, this paper presents a self-supervised deep learning approach to general image restoration problems. The key ingredient of the neuralized estimator is an adaptive stochastic gradient Langevin dy-namics algorithm for efficiently sampling the posterior distri-bution of network weights. The proposed method is applied on two image restoration problems: compressed sensing and phase retrieval. The experiments on these applications showed that the proposed method not only outperformed existing non-learning and unsupervised solutions in terms of image restoration quality, but also is more computationally efficient.
BibTeX
@inproceedings{wang2022self,
  author = {Weixi Wang and Ji Li and Hui Ji},
  title = {{Self-supervised Deep Image Restoration via Adaptive Stochastic Gradient Langevin Dynamics}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00203},
}
Figure