Abstract
Diffusion models are prominent in image generation for producing detailed and realistic images from Gaussian noises. However, they often encounter instability issues in image restoration tasks, e.g., super-resolution. Existing methods typically rely on multiple runs to find an initial noise that produces a reasonably restored image. Unfortunately, these methods are computationally expensive and time-consuming without guaranteeing stable and consistent performance. To address these challenges, we propose a novel Predictive Noise Fusion Strategy (PNFS) that predicts pixel-wise errors in the restored image and combines different noises to generate a more effective noise. Extensive experiments show that PNFS significantly improves the stability and performance of diffusion models in super-resolution, both quantitatively and qualitatively. Furthermore, PNFS can be flexibly integrated into various diffusion models to enhance their stability.
Introduction
While Diffusion Models (DMs) excel at generating realistic details, they struggle with the "ill-posed" nature of Super-Resolution (SR). In SR, a single low-resolution (LR) input can correspond to multiple valid high-resolution (HR) outputs. Current DM-based SR methods rely on starting from Gaussian noise; however, the choice of the initial random seed significantly impacts the final result.
The authors observe that different random seeds can lead to performance fluctuations of over 4 dB in PSNR. One seed might yield a near-perfect reconstruction, while another produces "hallucinated" artifacts or blurriness. Currently, practitioners often resort to multiple runs to find a "good" result, which is computationally inefficient and impractical for real-world deployment.
Methodology: Predictive Noise Fusion Strategy (PNFS)
To resolve this instability, the authors introduce the Predictive Noise Fusion Strategy (PNFS). This plug-and-play method acts as a pre-processor for the initial noise and consists of two primary modules:
- Discrepancy Prediction Module (DPM): A lightweight Transformer-based network that takes the LR image and an initial Gaussian noise map as input. It predicts a "discrepancy matrix," estimating which pixels in the final output are likely to exhibit high content or perceptual errors for that specific noise seed.
- Probabilistic Fusion Module (PFM): Instead of selecting a single "best" noise map, the PFM fuses elements from multiple noise maps. It uses the discrepancy matrices to create a probability map and employs multinomial sampling to select the optimal noise elements for each pixel.
Crucially, the PFM’s sampling strategy ensures the fused noise still follows a standard Gaussian distribution $N(0, I)$. This maintains the mathematical integrity required by pre-trained diffusion models while providing a superior starting point.
Experimental Results
The researchers evaluated PNFS on standard benchmarks (Set5, Set14, BSD100, Urban100, Manga109) and real-world datasets (RealSR). The DPM was trained on 900 images from DIV2K and Flickr2K.
Quantitative and Qualitative Performance
PNFS demonstrates significant improvements in both stability and image quality. On the Manga109 dataset, it improved the PSNR of the LDM backbone by 0.44 dB and the SD-Upscaler by 0.89 dB. In "worst-case" scenarios where standard seeds failed, PNFS uplifted PSNR by as much as 4.42 dB.
As shown in the figure above, PNFS (labeled "Ours") consistently maintains higher PSNR across different noise indices compared to the baseline LDM and SD-Upscaler. Qualitatively, PNFS eliminates distorted text and warped textures, producing results much closer to the ground truth than individual noise seeds.
Stability and Efficiency
The paper introduces the Local Standard Deviation (L-STD) metric to quantify stability. While traditional models exhibit high L-STD due to inconsistency, PNFS achieves a very low L-STD (~0.02), indicating highly predictable performance. Furthermore, the DPM is highly efficient, with a size of only 13.69 MB (approx. 0.1% of the backbone model) and adding a negligible 0.03 seconds to inference time.
Conclusion
PNFS provides a robust, lightweight solution to the instability inherent in diffusion-based super-resolution. By predicting and fusing noise locally, it ensures high-quality, consistent results without the need for expensive retraining of the diffusion backbone. Future work may explore the application of this predictive fusion strategy to other inverse problems such as image inpainting or deblurring.
Summary updated