Abstract
Diffusion Models (DMs) have revolutionized image super-resolution by generating highly realistic textures, yet they suffer from significant content inconsistency and high computational costs. We present Content Consistent Super-Resolution (CCSR), a framework that partitions the restoration process into structure refinement and detail enhancement. By utilizing a non-uniform sampling strategy and a GAN-finetuned VAE decoder, CCSR produces stable, high-quality results in as few as 1 or 2 steps, effectively eliminating the stochastic "hallucinations" common in iterative diffusion methods.
Introduction
Image Super-Resolution (SR) involves recovering high-resolution (HR) details from low-resolution (LR) inputs. While DMs excel at synthesizing realistic textures, their reliance on iterative noise sampling leads to two primary issues: instability and inefficiency. The same LR input can yield different HR outputs depending on the random seed, a phenomenon known as content inconsistency. Furthermore, the requirement for hundreds of sampling steps makes them impractical for real-time applications.
Previous GAN-based methods offer speed and stability but often fail to reconstruct complex natural structures. CCSR bridges this gap by combining the structural generative power of DMs with the deterministic efficiency of Generative Adversarial Networks (GANs).
Methodology: Disentangling Structure and Detail
The CCSR framework adopts a two-stage pipeline to separate the generation of macro-structures from the synthesis of fine-grained details.
Stage 1: Structure Refinement
DMs are particularly effective at reconstructing an image's underlying geometry during the early stages of the reverse diffusion process. CCSR introduces Non-Uniform Timestep Sampling (NUTS). Instead of starting from pure Gaussian noise, the model extracts coarse information from the LR image at a specific timestep and performs a limited number of reverse steps to solidify the primary structure. The process is truncated early to prevent the introduction of random, unfaithful textures that typically arise in later diffusion stages.
Stage 2: Detail Enhancement
To add fine textures without the randomness of diffusion, the authors finetune the pre-trained VAE decoder of the Stable Diffusion model using an adversarial (GAN) loss. This allows the model to synthesize details deterministically based on the latent features produced in Stage 1. This approach avoids additional computational overhead during inference while ensuring that the generated details are consistent with the refined structure.
Experimental Results
CCSR was evaluated on real-world datasets (RealSR, DRealSR) and synthetic benchmarks (DIV2K).
Stability and Efficiency
A key contribution of CCSR is its "content consistency." The authors introduced Global and Local Standard Deviation (G-STD, L-STD) metrics to quantify output variance across different random seeds. CCSR demonstrates significantly lower variance compared to existing DM-based methods like StableSR or SeeSR.
As shown in Figure 1, while other models produce varying textures for the same input, CCSR's outputs remain stable. Moreover, CCSR achieves competitive perceptual quality in just 1 or 2 steps, whereas baseline models often require 50 or more.
Perceptual Quality
In quantitative benchmarks such as MUSIQ and CLIPIQA, CCSR-S2 (the 2-step variant) consistently ranks among the top performers. It strikes a superior balance between fidelity (PSNR) and visual appeal, avoiding the over-smoothing of GANs and the over-generation of standard DMs.
Conclusion
CCSR addresses the core weaknesses of diffusion-based super-resolution by strategically combining DM and GAN strengths. By truncating the diffusion process and offloading detail synthesis to a deterministic GAN-based decoder, the framework achieves state-of-the-art performance with unprecedented stability and efficiency. Future work will focus on further optimizing the perception-fidelity trade-off to minimize the gap between generated quality and ground-truth accuracy.
Summary updated