Introduction
Deep learning has significantly advanced Single Image Super-Resolution (SISR), yet the internal mechanisms of these "black box" networks remain largely opaque. While researchers continue to propose deeper architectures and complex attention modules, it is often unclear whether these innovations truly enhance the network's ability to utilize spatial information or merely increase parameter redundancy. This paper introduces Local Attribution Maps (LAM), a diagnostic tool designed to reveal which input pixels contribute to the reconstruction of specific output patches. By quantifying this information usage through a novel Diffusion Index (DI), the authors demonstrate that a model's capacity to integrate a wider range of input pixels is a primary driver of superior SR performance.
The Interpretability Gap in Super-Resolution
Standard interpretability tools, typically developed for image classification, are ill-suited for SR tasks. In classification, the goal is to identify global features (e.g., "what makes this a cat"). In contrast, SR is a dense, local task where the objective is to understand how surrounding pixels contribute to the restoration of fine textures or sharp edges. LAM addresses this by visualizing and quantifying the "area of contribution" for any given SR model.
Methodology: Local Attribution Maps (LAM)
The proposed LAM method builds upon Integrated Gradients but is specifically tailored for the constraints of super-resolution.
The method involves four key components:
- Local Patch Attribution: LAM focuses on the attribution of a specific $l \times l$ patch in the output rather than the entire image.
- Blurred Baseline: Unlike classification methods that use a black image as a baseline, LAM utilizes a blurred version of the input. This represents the "absence" of high-frequency details that the network aims to recover.
- Progressive Blurring Path: To calculate attribution, the method transitions from the blurred baseline to the sharp input along a path that follows the natural distribution of images.
- Diffusion Index (DI): To quantify the visual maps, the authors introduce the DI. Based on the Gini coefficient, the DI measures how "spread out" a network's attention is. A higher DI indicates that the network effectively gathers information from a broader spatial range.
Key Insights and Results
The authors evaluated 48 different SR networks using benchmarks like DIV2K and Urban100, alongside a curated set of 150 challenging patches.
Several critical insights emerged from this analysis:
- Receptive Field vs. Contribution Area: A larger theoretical receptive field does not necessarily translate to a larger actual contribution area. LAM reveals that for many deep networks, the pixels that truly influence the output are confined to a much smaller region than theoretically possible.
- The "Range" Rule: There is a strong positive correlation (Pearson > 0.8) between the Diffusion Index and SR performance (PSNR/LPIPS). The most effective networks are those that "hunt" for information across a wider spatial range.
- Texture vs. Semantics: SR networks are highly sensitive to regular, repeating patterns (e.g., architectural grids) but struggle to utilize high-level semantic features (e.g., facial components) for reconstruction.
- Effectiveness of Attention: Non-local and attention-based networks (e.g., RNAN, SAN) outperform standard CNNs like EDSR because they significantly expand the range of involved pixels, as evidenced by the "sparks" of attribution in LAM maps.
Conclusion
LAM provides a powerful diagnostic framework for understanding SR networks. By showing that performance is intrinsically linked to the spatial extent of utilized information, this work offers a new metric for architecture design. Future research may leverage LAM to bridge the gap between low-level texture synthesis and high-level semantic understanding, potentially leading to more "intelligent" super-resolution models.
Summary updated