Accelerating Diffusion-based Super-Resolution with Dynamic Time-Spatial Sampling

Rui Qin, Qijie Wang, Ming Sun, Haowei Zhu, Chao Zhou, Bin Wang

Abstract

Diffusion models have gained attention for their success in modeling complex distributions, achieving impressive perceptual quality in SR tasks. However, existing diffusion-based SR methods often suffer from high computational costs, requiring numerous iterative steps for training and inference. Existing acceleration techniques, such as distillation and solver optimization, are generally task-agnostic and do not fully leverage the specific characteristics of low-level tasks like super-resolution (SR). In this study, we analyze the frequency- and spatial-domain properties of diffusion-based SR methods, revealing key insights into the temporal and spatial dependencies of high-frequency signal recovery. Specifically, high-frequency details benefit from concentrated optimization during early and late diffusion iterations, while spatially textured regions demand adaptive denoising strategies. Building on these observations, we propose the Time-Spatial-aware Sampling strategy (TSS) for the acceleration of Diffusion SR without any extra training cost. TSS combines Time Dynamic Sampling (TDS), which allocates more iterations to refining textures, and Spatial Dynamic Sampling (SDS), which dynamically adjusts strategies based on image content. Extensive evaluations across multiple benchmarks demonstrate that TSS achieves state-of-the-art (SOTA) performance with significantly fewer iterations, improving MUSIQ scores by 0.2 - 3.0 and outperforming the current acceleration methods with only half the number of steps.

Introduction

While Diffusion Models (DMs) excel at reconstructing high-fidelity textures in super-resolution (SR) tasks, their iterative nature makes them computationally expensive for real-time applications. Existing acceleration techniques like distillation or solver optimization (e.g., DDIM) are generally task-agnostic, treating every timestep and every pixel with equal importance. For low-level vision tasks like SR, this uniform approach is inefficient. High-frequency (HF) details—the fine textures and sharp edges—do not recover at a constant rate throughout the diffusion process, nor are they distributed evenly across an image. Smooth areas and complex textures have fundamentally different denoising requirements.

Methodology: Time-Spatial-aware Sampling (TSS)

The proposed TSS strategy is a training-free approach that can be integrated into pre-trained diffusion models. It consists of two core components:

Time Dynamic Sampling (TDS)

Frequency-domain analysis reveals that HF details benefit most from concentrated optimization during the very early and very late stages of the diffusion process. TDS replaces uniform sampling with a non-uniform schedule using a two-stage polynomial function. This "stretches" the schedule to allocate more iterations to the start and end of the process, where texture refinement is most active.

Temporal analysis of high-frequency signal recovery and visual denoising progress across timesteps.

Spatial Dynamic Sampling (SDS)

Recognizing that different regions of an image need different levels of attention, SDS uses a local variance map to identify high-texture areas. It then adaptively adjusts the sampling frequency per-pixel. High-variance regions undergo more "inhomogeneous" sampling to boost HF recovery, while smooth regions are sampled more uniformly to maintain stability. To implement per-pixel timesteps without breaking the underlying neural network, spatially-varying time embeddings are injected directly into the main branch features.

Key Insights and Analysis

The development of TSS is grounded in several key observations regarding the diffusion process in SR:

  • The "U-Shaped" HF Recovery: High-frequency signal-to-noise ratio (SNR) improvements are most pronounced at the beginning (steps 1000–700) and the end (steps 400–0) of inference. The middle stages are less productive for texture restoration.
  • Texture-Specific Denoising: Smooth regions undergo visible denoising much earlier than textured regions. By the time a textured region starts to resolve, a smooth region might already be "finished," making a one-size-fits-all sampling rate wasteful.
High-frequency SNR analysis for image patches categorized by local variance (High, Medium, Low).

Experimental Results

TSS was evaluated across multiple benchmarks, including DIV2K, RealSR, DRealSR, and RealPhoto60.

  • Efficiency vs. Quality: SUPIR equipped with TSS achieved better perceptual results in 7 steps than the baseline did in 14 or even 50 steps.
  • Quantitative Gains: TSS consistently improved MUSIQ scores by 0.2 to 3.0 across various benchmarks. On the RealPhoto60 dataset, TSS outperformed specialized accelerated models like ReShift and SinSR while using significantly fewer iterations and requiring zero extra training cost.
  • Portability: As a sampling strategy, TSS is highly portable and was successfully integrated with StableSR, SUPIR, and PASD, yielding performance gains in all cases.

Conclusion

TSS offers a powerful, training-free method to accelerate diffusion-based SR. By leveraging the temporal and spatial characteristics of high-frequency signal recovery, it achieves state-of-the-art performance with significantly reduced computational cost. Future work will explore applying TSS to other tasks like image inpainting and video super-resolution.

Summary updated

Abstract

Diffusion models have gained attention for their success in modeling complex distributions, achieving impressive perceptual quality in SR tasks. However, existing diffusion-based SR methods often suffer from high computational costs, requiring numerous iterative steps for training and inference. Existing acceleration techniques, such as distillation and solver optimization, are generally task-agnostic and do not fully leverage the specific characteristics of low-level tasks like super-resolution (SR). In this study, we analyze the frequency- and spatial-domain properties of diffusion-based SR methods, revealing key insights into the temporal and spatial dependencies of high-frequency signal recovery. Specifically, high-frequency details benefit from concentrated optimization during early and late diffusion iterations, while spatially textured regions demand adaptive denoising strategies. Building on these observations, we propose the Time-Spatial-aware Sampling strategy (TSS) for the acceleration of Diffusion SR without any extra training cost. TSS combines Time Dynamic Sampling (TDS), which allocates more iterations to refining textures, and Spatial Dynamic Sampling (SDS), which dynamically adjusts strategies based on image content. Extensive evaluations across multiple benchmarks demonstrate that TSS achieves state-of-the-art (SOTA) performance with significantly fewer iterations, improving MUSIQ scores by 0.2 - 3.0 and outperforming the current acceleration methods with only half the number of steps.
BibTeX
@inproceedings{qin2025accelerating,
  author = {Rui Qin and Qijie Wang and Ming Sun and Haowei Zhu and Chao Zhou and Bin Wang},
  title = {{Accelerating Diffusion-based Super-Resolution with Dynamic Time-Spatial Sampling}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=F1dHriJB3K},
}
Figure