Abstract
Modern Single Image Super-Resolution (SISR) models face a significant trade-off between perceptual quality and computational efficiency. While state-of-the-art generative models produce high-quality textures, they are often too heavy for real-time applications. Conversely, lightweight models typically suffer from blurriness due to their reliance on spatial pixel-wise losses. This paper introduces a novel training framework utilizing Fourier space losses to bridge this gap. By supervising lightweight models in the frequency domain, we achieve perceptual quality comparable to heavy state-of-the-art architectures while operating up to 48x faster.
Introduction
The goal of SISR is to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. Current state-of-the-art (SOTA) models like SRFlow and ESRGAN achieve impressive results but are computationally prohibitive for mobile or real-time use. Smaller, more efficient models often fall into the "blurriness trap," where standard spatial losses (L1/L2) lead to blurry outputs that lack high-frequency details like textures and sharp edges. This research explores whether a lightweight model can achieve SOTA perceptual performance through improved supervision strategies during training.
Methodology: Frequency-Domain Supervision
The core innovation lies in shifting supervision from the spatial domain to the Fourier domain. This approach provides global guidance that local pixel-wise losses lack.
Fourier Space Supervision Loss ($L_F$)
Instead of comparing images solely pixel-by-pixel, the proposed method applies a Fast Fourier Transform (FFT) to both generated and ground-truth images. The loss is calculated based on differences in amplitude and phase across frequency bands. This forces the generator to explicitly restore high-frequency components often lost during downsampling.
Fourier GAN Loss ($L_{GAN}^F$)
Generative Adversarial Networks (GANs) typically use convolutional neural network (CNN) discriminators. However, since spatial invariance does not apply in Fourier space, the authors designed a fully connected (FC) discriminator. This discriminator operates directly on Fourier coefficients to guide the hallucination of realistic textures.
Experimental Results
The models were trained on the DF2K dataset and evaluated on standard benchmarks including DIV2K, Urban100, and BSD100. The researchers used the Information Multi-Distillation Network (IMDN) as a lightweight backbone to demonstrate the effectiveness of their loss functions.
Performance and Efficiency
The "Full" model demonstrates a massive leap in efficiency:
- Speed: It is 2.4x faster than RankSRGAN and 48x faster than SRFlow on a GPU.
- Perceptual Quality: Despite its small size, the model achieves LPIPS scores comparable to much larger models like ESRGAN.
- Generalization: Adding Fourier losses to the standard ESRGAN training pipeline further improved its performance (+0.44dB PSNR and lower FID), indicating the broad utility of frequency-domain supervision.
Conclusion
By leveraging Fourier space losses, lightweight super-resolution models can overcome the limitations of spatial-only training. This approach allows smaller networks to utilize their representative power more effectively, achieving high-fidelity perceptual results at a fraction of the computational cost. Future work may extend these frequency-domain techniques to other restoration tasks such as image deblurring and denoising.
Summary updated