Introduction
In the field of "blind" Super-Resolution (SR), models must enhance images without prior knowledge of the specific degradation process (e.g., blur kernels or noise levels). State-of-the-art approaches often rely on Non-Local Attention (NLA) to capture self-similarity and estimate these degradations. However, NLA suffers from a significant computational bottleneck: the Softmax function. Standard attention requires the computation of an $N \times N$ attention map (where $N$ is the number of pixels), leading to quadratic $O(N^2)$ complexity. This makes high-resolution processing prohibitively slow and memory-intensive for edge devices.
Proposed Methodology
This paper introduces a highly efficient framework that replaces standard attention with a Second-Order Taylor Expansion Approximation (STEA) and pairs it with a Multi-scale Large Field Reception (MLFR) module. Together, these innovations achieve linear $O(N)$ complexity while maintaining high restoration quality.
1. Second-Order Taylor Expansion Approximation (STEA)
The core innovation involves mathematically rewriting the Softmax-based attention. By approximating the exponential function within Softmax using a second-order Taylor series, the authors eliminate the need for the massive $N \times N$ matrix. This allows the model to leverage the associative property of matrix multiplication, reordering operations to achieve linear complexity.
2. Multi-Scale Large Field Reception (MLFR)
Because approximating attention can reduce global context, the MLFR module is designed to compensate. It utilizes multiple branches of dilated convolutions with varying kernels and dilation rates. This ensures the model maintains a wide receptive field, capturing sufficient spatial information for accurate restoration.
Architecture: LabNet and RealNet
The authors integrated these modules into two distinct architectures:
- LabNet: Optimized for standard synthetic benchmarks, focusing on deep feature extraction through LSTEA modules.
- RealNet: Designed for real-world images, incorporating a training pipeline with specific degradations (noise and blur). RealNet features an Adapter and Fusion module with adjustable parameters ($\alpha$ and $\beta$), allowing users to manually balance denoising and deblurring based on the specific image content.
Key Results and Insights
Efficiency and Performance
- Speed: LabNet is nearly twice as fast as previous state-of-the-art models like NLCUNet (1.18s vs 2.23s for a $4\times$ upscale). Within the RealNet framework, the STEA module is approximately 10x faster than standard NLA (0.48s vs 4.21s).
- Computational Load: LabNet requires only 49 GFLOPs, a massive reduction from the 432 GFLOPs required by NLCUNet.
- Quality: On synthetic benchmarks like Urban100 ($x2$ scale), LabNet achieved a PSNR improvement of 0.11 dB over NLCUNet while remaining significantly lighter.
Real-World Restoration
RealNet demonstrates superior color coherence and detail restoration on the RealWorld38 dataset. Unlike "black box" models, the inclusion of adjustable fusion parameters acknowledges the subjective nature of real-world degradation, giving users control over the final aesthetic.
Conclusion
By breaking the "Softmax barrier" through Taylor expansion, this work provides a scalable path for high-performance super-resolution on resource-constrained hardware. While the Taylor expansion is an approximation, the addition of the MLFR module effectively bridges the performance gap, offering a state-of-the-art balance between speed and visual fidelity.
Summary updated