Abstract
Generative diffusion models have established a new state-of-the-art in Image Super-Resolution (SR), yet they remain computationally expensive and prone to artifacts in low-complexity regions. Furthermore, standard diffusion architectures often exhibit severe training instabilities, such as global color shifting, when trained on small batch sizes. We introduce YODA ("You Only Diffuse Areas"), a novel framework that optimizes the diffusion process by focusing exclusively on salient image regions. By employing dynamic, time-dependent attention masks derived from self-supervised DINO features, YODA concentrates refinement on high-frequency structural details while preserving the integrity of smooth backgrounds. Our approach significantly stabilizes training in resource-constrained environments and achieves substantial improvements in both objective metrics (PSNR) and perceptual quality (LPIPS) across multiple SR benchmarks.
Introduction
Image Super-Resolution (SR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. While generative diffusion models excel at synthesizing realistic textures, they typically treat every pixel with equal importance. This uniform processing is inefficient; a model spends as much effort denoising a flat, monochromatic sky as it does a highly detailed human eye.
This "one-size-fits-all" approach introduces two primary challenges. First, it often leads to the "hallucination" of unwanted textures or noise in simple backgrounds. Second, leading models like SR3 require massive batch sizes (e.g., 256) to maintain color accuracy. On consumer-grade hardware where only small batches are feasible, these models suffer from significant "color shifting," where the output's global tint deviates sharply from the LR source. YODA addresses these issues by introducing a spatially-aware diffusion mechanism.
Methodology: The YODA Framework
The YODA framework operates through a three-phase guided refinement process designed to focus computational energy where it matters most.
1. Identifying Key Regions
Rather than relying on manual saliency labels, YODA utilizes features from DINO, a self-supervised vision transformer. By extracting attention maps from the LR input, the system identifies "salient" regions—areas containing critical structural information like facial features or complex object textures.
2. Time-Dependent Masking
A core innovation of YODA is the use of dynamic rather than static masks. At the beginning of the reverse diffusion process (high noise levels), the mask is highly restrictive, focusing only on the most detail-rich areas. As the process approaches the final image (low noise), the mask progressively expands to cover the entire image. This ensures that complex regions receive more refinement steps than simple ones, preventing the introduction of artifacts in smooth areas early in the process.
3. Guided Backward Diffusion
During each denoising step, the model only updates pixels within the current mask. Pixels outside the mask are maintained as noisy versions of the LR input, and the two regions are blended seamlessly. Crucially, the optimization loss function is modified to be "mask-aware," penalizing the model only for errors within the active refinement region.
Experimental Results
Performance Gains and Stability
The researchers integrated YODA into several prominent architectures, including SR3, DiffBIR, and SRDiff. The most dramatic improvements were observed in Face SR tasks. For 4x upscaling on the CelebA-HQ dataset, adding YODA to the SR3 baseline increased the PSNR from 17.98 to 26.33 and improved the LPIPS perceptual score from 0.138 to 0.090.
Beyond raw metrics, YODA effectively solved the color shift problem. While standard SR3 models trained with a batch size of 8 produced distorted colors, YODA's selective focus anchored the global color distribution to the LR input, enabling high-quality training on limited hardware.
Attention Strategy Comparison
The study compared various saliency extraction methods, including non-learnable innate methods (Gaussian, Edge-based, SIFT) and learnable DINO-based features. As shown in the comparative results, the DINO with ResNet-50 (Attention-MAX) configuration yielded the best performance across all metrics, confirming that self-supervised features are excellent proxies for identifying refinement-heavy regions.
Conclusion and Future Work
YODA demonstrates that "less is more" in generative diffusion for SR. By selectively refining salient regions, the model produces cleaner backgrounds and sharper textures while stabilizing the training process against color distortions. While the current implementation relies on DINO's fixed resolution, future work may explore scale-invariant attention extraction and the integration of YODA into latent diffusion models to further drive down computational costs. Qualitatively and quantitatively, YODA represents a significant step toward more efficient and robust generative super-resolution.
Summary updated