Improving the Stability of Diffusion Models for Content Consistent Super-Resolution

Lingchen Sun, Rongyuan Wu, Zhengqiang Zhang, Hongwei Yong, Lei Zhang

Abstract

Diffusion Models (DMs) have revolutionized image super-resolution by generating highly realistic textures, yet they suffer from significant content inconsistency and high computational costs. We present Content Consistent Super-Resolution (CCSR), a framework that partitions the restoration process into structure refinement and detail enhancement. By utilizing a non-uniform sampling strategy and a GAN-finetuned VAE decoder, CCSR produces stable, high-quality results in as few as 1 or 2 steps, effectively eliminating the stochastic "hallucinations" common in iterative diffusion methods.

Introduction

Image Super-Resolution (SR) involves recovering high-resolution (HR) details from low-resolution (LR) inputs. While DMs excel at synthesizing realistic textures, their reliance on iterative noise sampling leads to two primary issues: instability and inefficiency. The same LR input can yield different HR outputs depending on the random seed, a phenomenon known as content inconsistency. Furthermore, the requirement for hundreds of sampling steps makes them impractical for real-time applications.

Previous GAN-based methods offer speed and stability but often fail to reconstruct complex natural structures. CCSR bridges this gap by combining the structural generative power of DMs with the deterministic efficiency of Generative Adversarial Networks (GANs).

Methodology: Disentangling Structure and Detail

The CCSR framework adopts a two-stage pipeline to separate the generation of macro-structures from the synthesis of fine-grained details.

The CCSR architecture features a two-stage process: Stage 1 focuses on structure refinement using a truncated diffusion process, while Stage 2 enhances details through a finetuned VAE decoder.

Stage 1: Structure Refinement

DMs are particularly effective at reconstructing an image's underlying geometry during the early stages of the reverse diffusion process. CCSR introduces Non-Uniform Timestep Sampling (NUTS). Instead of starting from pure Gaussian noise, the model extracts coarse information from the LR image at a specific timestep and performs a limited number of reverse steps to solidify the primary structure. The process is truncated early to prevent the introduction of random, unfaithful textures that typically arise in later diffusion stages.

Stage 2: Detail Enhancement

To add fine textures without the randomness of diffusion, the authors finetune the pre-trained VAE decoder of the Stable Diffusion model using an adversarial (GAN) loss. This allows the model to synthesize details deterministically based on the latent features produced in Stage 1. This approach avoids additional computational overhead during inference while ensuring that the generated details are consistent with the refined structure.

Experimental Results

CCSR was evaluated on real-world datasets (RealSR, DRealSR) and synthetic benchmarks (DIV2K).

Stability and Efficiency

A key contribution of CCSR is its "content consistency." The authors introduced Global and Local Standard Deviation (G-STD, L-STD) metrics to quantify output variance across different random seeds. CCSR demonstrates significantly lower variance compared to existing DM-based methods like StableSR or SeeSR.

Visual comparison of CCSR against other diffusion-based SR methods. CCSR maintains consistent content across different random seeds (S1-1 vs S1-2) while requiring significantly fewer steps.

As shown in Figure 1, while other models produce varying textures for the same input, CCSR's outputs remain stable. Moreover, CCSR achieves competitive perceptual quality in just 1 or 2 steps, whereas baseline models often require 50 or more.

Perceptual Quality

In quantitative benchmarks such as MUSIQ and CLIPIQA, CCSR-S2 (the 2-step variant) consistently ranks among the top performers. It strikes a superior balance between fidelity (PSNR) and visual appeal, avoiding the over-smoothing of GANs and the over-generation of standard DMs.

Conclusion

CCSR addresses the core weaknesses of diffusion-based super-resolution by strategically combining DM and GAN strengths. By truncating the diffusion process and offloading detail synthesis to a deterministic GAN-based decoder, the framework achieves state-of-the-art performance with unprecedented stability and efficiency. Future work will focus on further optimizing the perception-fidelity trade-off to minimize the gap between generated quality and ground-truth accuracy.

Summary updated

Abstract

The generative priors of pre-trained latent diffusion models (DMs) have demonstrated great potential to enhance the visual quality of image super-resolution (SR) results. However, the noise sampling process in DMs introduces randomness in the SR outputs, and the generated contents can differ a lot with different noise samples. The multi-step diffusion process can be accelerated by distilling methods, but the generative capacity is difficult to control. To address these issues, we analyze the respective advantages of DMs and generative adversarial networks (GANs) and propose to partition the generative SR process into two stages, where the DM is employed for reconstructing image structures and the GAN is employed for improving fine-grained details. Specifically, we propose a non-uniform timestep sampling strategy in the first stage. A single timestep sampling is first applied to extract the coarse information from the input image, then a few reverse steps are used to reconstruct the main structures. In the second stage, we finetune the decoder of the pre-trained variational auto-encoder by adversarial GAN training for deterministic detail enhancement. Once trained, our proposed method, namely content consistent super-resolution (CCSR),allows flexible use of different diffusion steps in the inference stage without re-training. Extensive experiments show that with 2 or even 1 diffusion step, CCSR can significantly improve the content consistency of SR outputs while keeping high perceptual quality. Codes and models can be found at \href{https://github.com/csslc/CCSR}{https://github.com/csslc/CCSR}.
BibTeX
@article{sun2025improving,
  author = {Lingchen Sun and Rongyuan Wu and Zhengqiang Zhang and Hongwei Yong and Lei Zhang},
  title = {{Improving the Stability of Diffusion Models for Content Consistent Super-Resolution}},
  journal = {IEEE Trans. Image Process.},
  year = {2025},
  url = {https://openreview.net/forum?id=bF2wTB1p4d},
}
Figure