Fourier Space Losses for Efficient Perceptual Image Super-Resolution

Dario Fuoli, Luc Van Gool, Radu Timofte

Abstract

Modern Single Image Super-Resolution (SISR) models face a significant trade-off between perceptual quality and computational efficiency. While state-of-the-art generative models produce high-quality textures, they are often too heavy for real-time applications. Conversely, lightweight models typically suffer from blurriness due to their reliance on spatial pixel-wise losses. This paper introduces a novel training framework utilizing Fourier space losses to bridge this gap. By supervising lightweight models in the frequency domain, we achieve perceptual quality comparable to heavy state-of-the-art architectures while operating up to 48x faster.

Introduction

The goal of SISR is to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. Current state-of-the-art (SOTA) models like SRFlow and ESRGAN achieve impressive results but are computationally prohibitive for mobile or real-time use. Smaller, more efficient models often fall into the "blurriness trap," where standard spatial losses (L1/L2) lead to blurry outputs that lack high-frequency details like textures and sharp edges. This research explores whether a lightweight model can achieve SOTA perceptual performance through improved supervision strategies during training.

Methodology: Frequency-Domain Supervision

The core innovation lies in shifting supervision from the spatial domain to the Fourier domain. This approach provides global guidance that local pixel-wise losses lack.

Overview of the proposed training framework featuring spatial and Fourier domain losses.

Fourier Space Supervision Loss ($L_F$)

Instead of comparing images solely pixel-by-pixel, the proposed method applies a Fast Fourier Transform (FFT) to both generated and ground-truth images. The loss is calculated based on differences in amplitude and phase across frequency bands. This forces the generator to explicitly restore high-frequency components often lost during downsampling.

Fourier GAN Loss ($L_{GAN}^F$)

Generative Adversarial Networks (GANs) typically use convolutional neural network (CNN) discriminators. However, since spatial invariance does not apply in Fourier space, the authors designed a fully connected (FC) discriminator. This discriminator operates directly on Fourier coefficients to guide the hallucination of realistic textures.

Experimental Results

The models were trained on the DF2K dataset and evaluated on standard benchmarks including DIV2K, Urban100, and BSD100. The researchers used the Information Multi-Distillation Network (IMDN) as a lightweight backbone to demonstrate the effectiveness of their loss functions.

Comparison of LPIPS (perceptual quality) vs. Runtime (ms). The proposed method achieves competitive quality with significantly lower latency.

Performance and Efficiency

The "Full" model demonstrates a massive leap in efficiency:

  • Speed: It is 2.4x faster than RankSRGAN and 48x faster than SRFlow on a GPU.
  • Perceptual Quality: Despite its small size, the model achieves LPIPS scores comparable to much larger models like ESRGAN.
  • Generalization: Adding Fourier losses to the standard ESRGAN training pipeline further improved its performance (+0.44dB PSNR and lower FID), indicating the broad utility of frequency-domain supervision.

Conclusion

By leveraging Fourier space losses, lightweight super-resolution models can overcome the limitations of spatial-only training. This approach allows smaller networks to utilize their representative power more effectively, achieving high-fidelity perceptual results at a fraction of the computational cost. Future work may extend these frequency-domain techniques to other restoration tasks such as image deblurring and denoising.

Summary updated

Abstract

Many super-resolution (SR) models are optimized for high performance only and therefore lack efficiency due to large model complexity. As large models are often not practical in real-world applications, we investigate and propose novel loss functions, to enable SR with high perceptual quality from much more efficient models. The representative power for a given low-complexity generator network can only be fully leveraged by strong guidance towards the optimal set of parameters. We show that it is possible to improve the performance of a recently introduced efficient generator architecture solely with the application of our proposed loss functions. In particular, we use a Fourier space supervision loss for improved restoration of missing high-frequency (HF) content from the ground truth image and design a discriminator architecture working directly in the Fourier domain to better match the target HF distribution. We show that our losses’ direct emphasis on the frequencies in Fourier-space significantly boosts the perceptual image quality, while at the same time retaining high restoration quality in comparison to previously proposed loss functions for this task. The performance is further improved by utilizing a combination of spatial and frequency domain losses, as both representations provide complementary information during training. On top of that, the trained generator achieves comparable results with and is 2.4× and 48× faster than state-of-the-art perceptual SR methods RankSRGAN and SRFlow respectively.
BibTeX
@misc{fuoli2021fourier,
  author = {Dario Fuoli and Luc Van Gool and Radu Timofte},
  title = {{Fourier Space Losses for Efficient Perceptual Image Super-Resolution}},
  year = {2021},
  eprint = {2106.00783},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.00783},
}
Figure