RELD: Regularization by Latent Diffusion Models for Image Restoration

Pasquale Cascarano, Lorenzo Stacchio, Andrea Sebastiani, Alessandro Benfenati, Ulugbek S. Kamilov, Gustavo MarfiaarXiv 2025 · DOI · OpenReview

TL;DR

This paper introduces RELD, a novel image restoration method that leverages Latent Diffusion Models (LDMs) within a variational framework. By training an LDM for denoising and integrating it into a Half-Quadratic Splitting optimization, RELD achieves competitive results in image restoration tasks like denoising, deblurring, and super-resolution, particularly excelling in perceptual quality.

What Problem Does This Paper Address?

Image restoration (IR) aims to reconstruct a high-quality image from degraded observations, often involving ill-posed inverse problems. Traditional methods rely on handcrafted regularizers, which struggle to capture the complexity of natural images. While data-driven approaches using deep neural networks have shown promise, they often require large training datasets and can be sensitive to out-of-distribution data, leading to artifacts. Existing methods that integrate generative models often bypass the theoretical underpinnings of variational optimization or face challenges in aligning denoisers with natural image manifolds. This paper addresses the need for a robust and efficient IR method that benefits from the generative power of diffusion models while maintaining a strong theoretical foundation and achieving high perceptual quality.

Methods

The core of RELD lies in its integration of Latent Diffusion Models (LDMs) into a variational framework using the Half-Quadratic Splitting (HQS) optimization method.

  • Latent Diffusion Models (LDMs): Instead of operating directly in the high-dimensional pixel space, LDMs work in a compressed latent space. This involves training an autoencoder to map images to a lower-dimensional representation and then training a denoising network (typically a U-Net) on this latent space. This reduces computational cost and allows the model to focus on semantic information.
  • Regularization by Latent Denoising (RELD): The LDM is trained specifically for the task of image denoising. This trained LDM then acts as a powerful, data-driven regularizer within the optimization process.
  • Variational Framework with HQS: The problem is formulated as minimizing an objective function that includes a data-fidelity term and a regularization term. HQS is employed to iteratively solve this optimization problem by splitting it into sub-problems.
  • Key Innovations:
    • Operating in Latent Space: Performing the denoising and optimization in the latent space of an LDM significantly reduces computational burden.
  • Warm-Start Initialization: The optimization process is initialized with a "warm-start" by incorporating information from the corrupted image b into the latent variable v.
  • Joint Optimization: Unlike some prior methods, RELD optimizes not only the latent variable but also the conditioning variables, leading to more robust alignment.
  • Conditional Generation: The LDM is trained to be a conditional generator, using the corrupted image b as conditioning information.

Datasets

  • Training: The LDM was trained on the Smartphone Image Denoising Dataset (SIDD) [68]. To adapt it for the LDM framework, patches were extracted from the high-resolution SIDD images, and synthetic Gaussian noise was added to create conditioning images.
  • Evaluation: The performance of RELD was evaluated on the Set5 dataset [69] for various image restoration tasks.

Key Insights and Ideas

  • LDMs as Powerful Regularizers: The paper demonstrates that LDMs, when trained for denoising, can serve as highly effective regularizers for inverse problems, capturing complex image priors better than traditional methods.
  • Latent Space Efficiency: Performing optimization in the latent space of an LDM offers a significant computational advantage without sacrificing restoration quality, making it more practical for high-resolution images.
  • Bridging Variational Methods and Generative Models: RELD successfully bridges the gap between theoretically grounded variational optimization (via HQS) and the powerful generative capabilities of diffusion models, offering a principled way to incorporate learned priors.
  • Perceptual Quality Advantage: While not always achieving the highest PSNR, RELD consistently produces more natural and visually appealing reconstructions, as evidenced by strong performance on perceptual metrics like NIQE and PIQE.

Key Results

  • Competitive Performance: RELD demonstrates competitive performance across denoising, deblurring, and super-resolution tasks when compared to state-of-the-art methods, including other diffusion-based approaches and Plug-and-Play (PnP) methods.
  • Superior Perceptual Quality: RELD consistently outperforms other methods in terms of perceptual quality metrics (NIQE, PIQE), producing sharper details and more natural textures, especially in challenging deblurring and super-resolution scenarios.
  • Deblurring: In deblurring experiments, RELD achieved the best results in NIQE and PIQE, and was highly competitive in LPIPS, outperforming DPS and DiffPIR in perceptual metrics.
  • Super-Resolution: For super-resolution, RELD significantly outperformed other diffusion-based approaches in PSNR and also showed strong performance in perceptual metrics, particularly in restoring finer details.
  • Computational Efficiency: Operating in the latent space of the LDM leads to reduced computational cost compared to image-space diffusion models.

Limitations and Future Work

  • Theoretical Foundation: The authors acknowledge that future work could involve developing a more rigorous theoretical foundation for the proposed method, particularly concerning the error induced by using a single gradient step in the optimization.
  • PSNR vs. Perceptual Quality: The paper notes that while RELD excels in perceptual quality, it may not always achieve the highest PSNR due to the nature of latent-space optimization, which doesn't strictly enforce pixel-level fidelity.
  • Hyperparameter Tuning: While the paper investigates hyperparameter sensitivity, further exploration into automatic parameter selection could be beneficial.

Summary updated

Abstract

In recent years, Diffusion Models have become the new state-of-the-art in deep generative modeling, ending the long-time dominance of Generative Adversarial Networks. Inspired by the Regularization by Denoising principle, we introduce an approach that integrates a Latent Diffusion Model, trained for the denoising task, into a variational framework using Half-Quadratic Splitting, exploiting its regularization properties. This approach, under appropriate conditions that can be easily met in various imaging applications, allows for reduced computational cost while achieving high-quality results. The proposed strategy, called Regularization by Latent Denoising (RELD), is then tested on a dataset of natural images, for image denoising, deblurring, and super-resolution tasks. The numerical experiments show that RELD is competitive with other state-of-the-art methods, particularly achieving remarkable results when evaluated using perceptual quality metrics.
BibTeX
@misc{cascarano2025reld,
  author = {Pasquale Cascarano and Lorenzo Stacchio and Andrea Sebastiani and Alessandro Benfenati and Ulugbek S. Kamilov and Gustavo Marfia},
  title = {{RELD: Regularization by Latent Diffusion Models for Image Restoration}},
  year = {2025},
  eprint = {2503.22563},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.22563},
}
Figure