Learning Diffusion Texture Priors for Image Restoration

Tian Ye, Sixiang Chen, Wenhao Chai, Zhaohu Xing, Jing Qin, Ge Lin, Lei Zhu

Abstract

Image restoration (IR) tasks such as deblurring, deraining, and desnowing often struggle with the "fidelity vs. realism" trade-off. Traditional regression models produce blurry results, while generative models frequently introduce hallucinations. This paper introduces the Diffusion Texture Prior Model (DTPM), a two-stage framework that leverages diffusion models to recover high-frequency textures while maintaining structural alignment. By separating the restoration of a deterministic structural "skeleton" from the generation of a stochastic texture prior, DTPM achieves state-of-the-art performance across multiple benchmarks with significantly improved perceptual quality.

Introduction

The challenge of image restoration lies in its ill-posed nature: a single degraded image can correspond to multiple plausible clean versions. Existing solutions generally fall into two categories. Regression-based models minimize pixel-level error (e.g., PSNR), leading to high fidelity but often "average" or blurry outputs lacking fine detail. Conversely, Generative models, particularly those based on GANs or Diffusion Models (DMs), produce visually stunning results but are prone to "hallucinations"—generating realistic textures that do not exist in the original scene.

The Diffusion Texture Prior Model (DTPM) addresses this by asking how diffusion can be constrained to generate realistic textures without sacrificing structural fidelity.

Methodology

The DTPM framework operates in two distinct stages: learning a general texture prior and then adapting it to specific restoration tasks.

Stage I: Learning the Texture Prior

Rather than training a diffusion model to generate entire images, the authors focus on the residual distribution. This represents the difference between a high-quality image and its smoothed, edge-preserved counterpart. By focusing purely on high-frequency details, the model becomes a specialized "texture engine." To ensure the generated textures are contextually appropriate (e.g., generating brick textures for walls rather than grass), the model incorporates Semantic Guidance via a pre-trained ResNet-18, which extracts global semantic codes to anchor the diffusion process.

Stage II: Task-Specific Adaptation

To apply the learned prior to tasks like deblurring or desnowing, DTPM utilizes a two-component system:

  1. Initial Predictor: A lightweight U-Net (DvSR) generates a "rough draft" of the restored image, establishing the structural skeleton.
  2. Conditional Guidance Adapters: Small, tunable layers are inserted into the frozen DTPM backbone. These adapters use the "rough draft" and the original degraded input to guide the diffusion process toward the final output.

This modular approach allows for efficient task adaptation without retraining the entire generative prior. Furthermore, by utilizing DDIM sampling, DTPM can produce high-quality results in as few as 4 to 10 steps, a significant speedup over standard diffusion methods.

Experimental Setup

The model was pre-trained on 55,558 high-quality patches from the DIV2K and Flickr2K datasets. Evaluation was conducted across a wide range of restoration benchmarks:

  • GoPro for motion deblurring.
  • DPDD for defocus deblurring.
  • Snow100K for desnowing.
  • Outdoor-Rain and Raindrop for deraining and dehazing.

Key Results

DTPM consistently outperforms state-of-the-art (SOTA) models like Restormer and NAFNet. In desnowing tasks on the Snow100K-S dataset, DTPM achieved a PSNR of 37.01, surpassing the previous best (SnowDiff) by nearly 0.5 dB. In deblurring on the DPDD dataset, the 4-step DTPM variant achieved superior LPIPS (0.153) and SSIM (0.823) scores.

Visual and metric progression of image restoration quality as the texture prior influence (alpha) increases.

As shown in the figure above, increasing the influence of the texture prior (represented by the parameter $\alpha$) leads to significant improvements in perceptual metrics such as NIQE and CLIP-IQA+. While PSNR may fluctuate slightly—a common phenomenon where "perfect" pixel alignment does not always equate to natural appearance—the visual fidelity of DTPM is markedly superior. Qualitative analysis shows the model's unique ability to restore legible text on license plates and intricate patterns on building facades where competing models often produce smudged or distorted outputs.

Discussion and Limitations

The core innovation of DTPM is the separation of concerns: a deterministic model handles the "where" (structure), while the diffusion model handles the "what" (texture). This prevents the stochastic nature of diffusion from creating "realistic but wrong" structures.

However, challenges remain. While 4-step sampling is fast for a diffusion model, it remains slower than "one-pass" regression models, limiting its current utility for real-time video applications. Additionally, the multi-component architecture (backbone, predictor, and adapters) results in a larger parameter count, which may require optimization for deployment on edge devices. Finally, performance can still degrade under extreme "blind" conditions where the initial predictor fails to provide a coherent structural skeleton.

Conclusion

The Diffusion Texture Prior Model represents a significant step forward in balancing fidelity and realism in image restoration. By treating diffusion as a specialized texture generator guided by structural and semantic constraints, DTPM provides a robust framework for high-quality, hallucination-free restoration across a variety of degradation types.

Summary updated

Abstract

Diffusion Models have shown remarkable performance in image generation tasks, which are capable of generating diverse and realistic image content. When adopting diffusion models for image restoration, the crucial challenge lies in how to preserve high-level image fidelity in the random-ness diffusion process and generate accurate background structures and realistic texture details. In this paper, we propose a general framework and develop a Diffusion Texture Prior Model (DTPM) for image restoration tasks. DTPM explicitly models high-quality texture details through the diffusion process, rather than global contextual content. In phase one of the training stage, we pretrain DTPM on approximately 55K high-quality image samples, after which we freeze most of its parameters. In phase two, we insert conditional guidance adapters into DTPM and equip it with an initial predictor, thereby facilitating its rapid adaptation to downstream image restoration tasks. Our DTPM could mitigate the randomness of traditional diffusion models by utilizing encapsulated rich and diverse texture knowledge and background structural information provided by the initial predictor during the sampling process.
BibTeX
@inproceedings{ye2024learning,
  author = {Tian Ye and Sixiang Chen and Wenhao Chai and Zhaohu Xing and Jing Qin and Ge Lin and Lei Zhu},
  title = {{Learning Diffusion Texture Priors for Image Restoration}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00244},
}
Figure