Dynamic Attention-Guided Diffusion for Image Super-Resolution

Brian B. Moser, Stanislav Frolov, Federico Raue, Sebastian Palacio, Andreas Dengel

Abstract

Generative diffusion models have established a new state-of-the-art in Image Super-Resolution (SR), yet they remain computationally expensive and prone to artifacts in low-complexity regions. Furthermore, standard diffusion architectures often exhibit severe training instabilities, such as global color shifting, when trained on small batch sizes. We introduce YODA ("You Only Diffuse Areas"), a novel framework that optimizes the diffusion process by focusing exclusively on salient image regions. By employing dynamic, time-dependent attention masks derived from self-supervised DINO features, YODA concentrates refinement on high-frequency structural details while preserving the integrity of smooth backgrounds. Our approach significantly stabilizes training in resource-constrained environments and achieves substantial improvements in both objective metrics (PSNR) and perceptual quality (LPIPS) across multiple SR benchmarks.

Introduction

Image Super-Resolution (SR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. While generative diffusion models excel at synthesizing realistic textures, they typically treat every pixel with equal importance. This uniform processing is inefficient; a model spends as much effort denoising a flat, monochromatic sky as it does a highly detailed human eye.

This "one-size-fits-all" approach introduces two primary challenges. First, it often leads to the "hallucination" of unwanted textures or noise in simple backgrounds. Second, leading models like SR3 require massive batch sizes (e.g., 256) to maintain color accuracy. On consumer-grade hardware where only small batches are feasible, these models suffer from significant "color shifting," where the output's global tint deviates sharply from the LR source. YODA addresses these issues by introducing a spatially-aware diffusion mechanism.

Methodology: The YODA Framework

The YODA framework operates through a three-phase guided refinement process designed to focus computational energy where it matters most.

1. Identifying Key Regions

Rather than relying on manual saliency labels, YODA utilizes features from DINO, a self-supervised vision transformer. By extracting attention maps from the LR input, the system identifies "salient" regions—areas containing critical structural information like facial features or complex object textures.

2. Time-Dependent Masking

A core innovation of YODA is the use of dynamic rather than static masks. At the beginning of the reverse diffusion process (high noise levels), the mask is highly restrictive, focusing only on the most detail-rich areas. As the process approaches the final image (low noise), the mask progressively expands to cover the entire image. This ensures that complex regions receive more refinement steps than simple ones, preventing the introduction of artifacts in smooth areas early in the process.

3. Guided Backward Diffusion

During each denoising step, the model only updates pixels within the current mask. Pixels outside the mask are maintained as noisy versions of the LR input, and the two regions are blended seamlessly. Crucially, the optimization loss function is modified to be "mask-aware," penalizing the model only for errors within the active refinement region.

The YODA framework overview, illustrating the three-phase process: identifying key regions, time-dependent masking, and guided backward diffusion.

Experimental Results

Performance Gains and Stability

The researchers integrated YODA into several prominent architectures, including SR3, DiffBIR, and SRDiff. The most dramatic improvements were observed in Face SR tasks. For 4x upscaling on the CelebA-HQ dataset, adding YODA to the SR3 baseline increased the PSNR from 17.98 to 26.33 and improved the LPIPS perceptual score from 0.138 to 0.090.

Beyond raw metrics, YODA effectively solved the color shift problem. While standard SR3 models trained with a batch size of 8 produced distorted colors, YODA's selective focus anchored the global color distribution to the LR input, enabling high-quality training on limited hardware.

Attention Strategy Comparison

The study compared various saliency extraction methods, including non-learnable innate methods (Gaussian, Edge-based, SIFT) and learnable DINO-based features. As shown in the comparative results, the DINO with ResNet-50 (Attention-MAX) configuration yielded the best performance across all metrics, confirming that self-supervised features are excellent proxies for identifying refinement-heavy regions.

Comparison of different attention map strategies and their quantitative impact on SR performance across various metrics.

Conclusion and Future Work

YODA demonstrates that "less is more" in generative diffusion for SR. By selectively refining salient regions, the model produces cleaner backgrounds and sharper textures while stabilizing the training process against color distortions. While the current implementation relies on DINO's fixed resolution, future work may explore scale-invariant attention extraction and the integration of YODA into latent diffusion models to further drive down computational costs. Qualitatively and quantitatively, YODA represents a significant step toward more efficient and robust generative super-resolution.

Summary updated

Abstract

Diffusion models in image Super-Resolution (SR) treat all image regions uniformly, which risks compromising the overall image quality by potentially introducing artifacts during denoising of less-complex regions. To address this, we propose “You Only Diffuse Areas” (YODA), a dynamic attention-guided diffusion process for image SR. YODA selectively focuses on spatial regions defined by attention maps derived from the low-resolution images and the current de-noising time step. This time-dependent targeting enables a more efficient conversion to high-resolution outputs by focusing on areas that benefit the most from the iterative refinement process, i.e., detail-rich objects. We empirically validate YODA by extending leading diffusion-based methods SR3, DiffBIR, and SRDiff. Our experiments demonstrate new state-of-the-art performances in face and general SR tasks across PSNR, SSIM, and LPIPS metrics. As a side effect, we find that YODA reduces color shift issues and stabilizes training with small batches.
BibTeX
@inproceedings{moser2025dynamic,
  author = {Brian B. Moser and Stanislav Frolov and Federico Raue and Sebastian Palacio and Andreas Dengel},
  title = {{Dynamic Attention-Guided Diffusion for Image Super-Resolution}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00054},
}
Figure