Accelerating Diffusion-based Super-Resolution with Dynamic Time-Spatial Sampling

Rui Qin, Qijie Wang, Ming Sun, Haowei Zhu, Chao Zhou, Bin Wang

Abstract

Diffusion models have gained attention for their success in modeling complex distributions, achieving impressive perceptual quality in SR tasks. However, existing diffusion-based SR methods often suffer from high computational costs, requiring numerous iterative steps for training and inference. Existing acceleration techniques, such as distillation and solver optimization, are generally task-agnostic and do not fully leverage the specific characteristics of low-level tasks like super-resolution (SR). In this study, we analyze the frequency- and spatial-domain properties of diffusion-based SR methods, revealing key insights into the temporal and spatial dependencies of high-frequency signal recovery. Specifically, high-frequency details benefit from concentrated optimization during early and late diffusion iterations, while spatially textured regions demand adaptive denoising strategies. Building on these observations, we propose the Time-Spatial-aware Sampling strategy (TSS) for the acceleration of Diffusion SR without any extra training cost. TSS combines Time Dynamic Sampling (TDS), which allocates more iterations to refining textures, and Spatial Dynamic Sampling (SDS), which dynamically adjusts strategies based on image content. Extensive evaluations across multiple benchmarks demonstrate that TSS achieves state-of-the-art (SOTA) performance with significantly fewer iterations, improving MUSIQ scores by 0.2~3.0 and outperforming the current acceleration methods with only half the number of steps.

Introduction

Diffusion models have revolutionized image super-resolution (SR) by generating highly realistic textures. However, their iterative nature makes them computationally expensive, often requiring dozens or hundreds of denoising steps to produce a single high-quality image. While general acceleration techniques like distillation or optimized solvers exist, they are typically task-agnostic, treating every pixel and timestep uniformly. This study argues that SR is unique because different image components—such as smooth backgrounds versus intricate textures—evolve at different rates during the denoising process. Ignoring these spatiotemporal dynamics leads to inefficient computation on "easy" regions and insufficient detail in complex ones.

Methodology

The proposed Time-Spatial-aware Sampling (TSS) is a training-free strategy consisting of two main components designed to exploit the unique characteristics of the SR task:

  1. Time Dynamic Sampling (TDS): Through frequency analysis, researchers found that high-frequency (HF) details are primarily recovered during the very early and very late stages of diffusion. TDS replaces uniform step-spacing with a non-uniform, polynomial-based schedule to allocate more iterations to these critical phases.
  2. Spatial Dynamic Sampling (SDS): SDS uses a local variance map to distinguish between smooth and textured regions. It adaptively assigns sampling densities: textured regions receive the specialized TDS treatment, while smooth regions are sampled more uniformly to prevent artifacts.
Signal-to-Noise Ratio (SNR) analysis during the diffusion process for different frequency signals and image patches.

To implement this efficiently, a Spatial Dynamic Time Embedding layer is introduced, allowing a single forward pass of the neural network to process different pixels at different "virtual" timesteps simultaneously, avoiding expensive image cropping.

Key Insights and Empirical Analysis

The core of TSS is built upon several empirical observations regarding the diffusion process in SR:

  • The Two-Stage Recovery of Detail: High-frequency signals follow a unique "U-shaped" recovery pattern, benefiting most from optimization at the very beginning and the very end of the denoising trajectory.
  • Content-Dependent Dynamics: Smooth regions (e.g., clear sky) reach a stable state much earlier than textured regions (e.g., animal fur). Using a uniform sampling rate across the whole image is computationally inefficient.
  • Training-Free Applicability: TSS is a "plug-and-play" strategy that can be integrated into existing pre-trained models like StableSR, SUPIR, and PASD without any additional training or fine-tuning cost.
Noise delta of high-frequency signals and visual denoising progress across different timesteps.

Experimental Evaluation

TSS was evaluated across synthetic (DIV2K) and real-world (RealSR, DRealSR, RealPhoto60) benchmarks, as well as face-specific datasets.

  • Efficiency vs. Quality: TSS-enhanced models outperformed SOTA acceleration methods like ResShift and SinSR using significantly fewer steps. For instance, SUPIR with TSS at 7 steps rivaled baseline models using 15+ steps.
  • Quantitative Gains: The method showed consistent improvements in perceptual metrics, boosting MUSIQ scores by 0.2 to 3.0 across various datasets.
  • Qualitative Superiority: Visually, TSS produces sharper and more realistic fine details—such as individual hairs or delicate wing patterns—where uniform sampling often leaves blurry textures or color-blocking artifacts.

Conclusion

TSS provides a significant leap in accelerating diffusion-based SR by leveraging task-specific spatiotemporal dynamics. While it relies on variance as a proxy for texture—which might be sensitive to extreme noise—it offers a robust, training-free solution for high-quality image generation. Future work may involve automating hyperparameter tuning based on input degradation or combining TSS with distillation methods for even greater efficiency.

Summary updated

Abstract

Diffusion models have gained attention for their success in modeling complex distributions, achieving impressive perceptual quality in SR tasks. However, existing diffusion-based SR methods often suffer from high computational costs, requiring numerous iterative steps for training and inference. Existing acceleration techniques, such as distillation and solver optimization, are generally task-agnostic and do not fully leverage the specific characteristics of low-level tasks like super-resolution (SR). In this study, we analyze the frequency- and spatial-domain properties of diffusion-based SR methods, revealing key insights into the temporal and spatial dependencies of high-frequency signal recovery. Specifically, high-frequency details benefit from concentrated optimization during early and late diffusion iterations, while spatially textured regions demand adaptive denoising strategies. Building on these observations, we propose the Time-Spatial-aware Sampling strategy (TSS) for the acceleration of Diffusion SR without any extra training cost. TSS combines Time Dynamic Sampling (TDS), which allocates more iterations to refining textures, and Spatial Dynamic Sampling (SDS), which dynamically adjusts strategies based on image content. Extensive evaluations across multiple benchmarks demonstrate that TSS achieves state-of-the-art (SOTA) performance with significantly fewer iterations, improving MUSIQ scores by 0.2~3.0 and outperforming the current acceleration methods with only half the number of steps.
BibTeX
@inproceedings{qin2025accelerating,
  author = {Rui Qin and Qijie Wang and Ming Sun and Haowei Zhu and Chao Zhou and Bin Wang},
  title = {{Accelerating Diffusion-based Super-Resolution with Dynamic Time-Spatial Sampling}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=F1dHriJB3K},
}
Figure