Abstract
Image restoration (IR) tasks such as deblurring, deraining, and desnowing often struggle with the "fidelity vs. realism" trade-off. Traditional regression models produce blurry results, while generative models frequently introduce hallucinations. This paper introduces the Diffusion Texture Prior Model (DTPM), a two-stage framework that leverages diffusion models to recover high-frequency textures while maintaining structural alignment. By separating the restoration of a deterministic structural "skeleton" from the generation of a stochastic texture prior, DTPM achieves state-of-the-art performance across multiple benchmarks with significantly improved perceptual quality.
Introduction
The challenge of image restoration lies in its ill-posed nature: a single degraded image can correspond to multiple plausible clean versions. Existing solutions generally fall into two categories. Regression-based models minimize pixel-level error (e.g., PSNR), leading to high fidelity but often "average" or blurry outputs lacking fine detail. Conversely, Generative models, particularly those based on GANs or Diffusion Models (DMs), produce visually stunning results but are prone to "hallucinations"—generating realistic textures that do not exist in the original scene.
The Diffusion Texture Prior Model (DTPM) addresses this by asking how diffusion can be constrained to generate realistic textures without sacrificing structural fidelity.
Methodology
The DTPM framework operates in two distinct stages: learning a general texture prior and then adapting it to specific restoration tasks.
Stage I: Learning the Texture Prior
Rather than training a diffusion model to generate entire images, the authors focus on the residual distribution. This represents the difference between a high-quality image and its smoothed, edge-preserved counterpart. By focusing purely on high-frequency details, the model becomes a specialized "texture engine." To ensure the generated textures are contextually appropriate (e.g., generating brick textures for walls rather than grass), the model incorporates Semantic Guidance via a pre-trained ResNet-18, which extracts global semantic codes to anchor the diffusion process.
Stage II: Task-Specific Adaptation
To apply the learned prior to tasks like deblurring or desnowing, DTPM utilizes a two-component system:
- Initial Predictor: A lightweight U-Net (DvSR) generates a "rough draft" of the restored image, establishing the structural skeleton.
- Conditional Guidance Adapters: Small, tunable layers are inserted into the frozen DTPM backbone. These adapters use the "rough draft" and the original degraded input to guide the diffusion process toward the final output.
This modular approach allows for efficient task adaptation without retraining the entire generative prior. Furthermore, by utilizing DDIM sampling, DTPM can produce high-quality results in as few as 4 to 10 steps, a significant speedup over standard diffusion methods.
Experimental Setup
The model was pre-trained on 55,558 high-quality patches from the DIV2K and Flickr2K datasets. Evaluation was conducted across a wide range of restoration benchmarks:
- GoPro for motion deblurring.
- DPDD for defocus deblurring.
- Snow100K for desnowing.
- Outdoor-Rain and Raindrop for deraining and dehazing.
Key Results
DTPM consistently outperforms state-of-the-art (SOTA) models like Restormer and NAFNet. In desnowing tasks on the Snow100K-S dataset, DTPM achieved a PSNR of 37.01, surpassing the previous best (SnowDiff) by nearly 0.5 dB. In deblurring on the DPDD dataset, the 4-step DTPM variant achieved superior LPIPS (0.153) and SSIM (0.823) scores.
As shown in the figure above, increasing the influence of the texture prior (represented by the parameter $\alpha$) leads to significant improvements in perceptual metrics such as NIQE and CLIP-IQA+. While PSNR may fluctuate slightly—a common phenomenon where "perfect" pixel alignment does not always equate to natural appearance—the visual fidelity of DTPM is markedly superior. Qualitative analysis shows the model's unique ability to restore legible text on license plates and intricate patterns on building facades where competing models often produce smudged or distorted outputs.
Discussion and Limitations
The core innovation of DTPM is the separation of concerns: a deterministic model handles the "where" (structure), while the diffusion model handles the "what" (texture). This prevents the stochastic nature of diffusion from creating "realistic but wrong" structures.
However, challenges remain. While 4-step sampling is fast for a diffusion model, it remains slower than "one-pass" regression models, limiting its current utility for real-time video applications. Additionally, the multi-component architecture (backbone, predictor, and adapters) results in a larger parameter count, which may require optimization for deployment on edge devices. Finally, performance can still degrade under extreme "blind" conditions where the initial predictor fails to provide a coherent structural skeleton.
Conclusion
The Diffusion Texture Prior Model represents a significant step forward in balancing fidelity and realism in image restoration. By treating diffusion as a specialized texture generator guided by structural and semantic constraints, DTPM provides a robust framework for high-quality, hallucination-free restoration across a variety of degradation types.
Summary updated