Deep Fractional Fourier Transform

Hu Yu, Jie Huang, Lingzhi Li, Man Zhou, Feng ZhaoNeurIPS 2023 · OpenReview

TL;DR

This paper introduces the Fractional Fourier Transform (FRFT) to deep learning, providing a unified framework that bridges the gap between spatial and frequency domains. By treating an image as a signal that can be rotated at any angle in the spatial-frequency plane, the authors develop a "Multi-order FRactional Fourier Convolution" (MFRFC) operator that significantly boosts performance across a wide array of vision tasks, from object detection to image restoration.

What Problem Does This Paper Address?

In computer vision, we typically operate in two "orthogonal" worlds: the spatial domain (standard convolutions) and the frequency domain (Fourier Transforms).

  • Spatial convolutions are great for extracting local features but have limited receptive fields.
  • Fourier Transforms (FT) provide a global receptive field but assume the signal is "stationary"—meaning its statistical properties don't change over space.

However, real-world images are non-stationary; they contain edges, textures, and structures that vary significantly across the frame. Standard FT captures the global spectrum but loses local spatial characteristics. The authors argue that we need a tool that doesn't just pick one domain or the other, but can provide an intermediate, unified perspective tailored for non-stationary signals.

Methods

The core contribution is the adaptation of the Fractional Fourier Transform (FRFT) for deep neural networks.

  1. The FRFT Operator: Think of the spatial and frequency domains as two axes on a graph. The FRFT allows you to rotate the signal by an arbitrary angle $\alpha = p\pi/2$. When $p=0$, you are entirely in the spatial domain; when $p=1$, you are entirely in the frequency domain. Any value in between provides a hybrid representation.
  2. MFRFC (Multi-order FRactional Fourier Convolution): The authors propose a plug-and-play module. It splits input features into three distinct paths:
  3. Spatial Path ($p=0$): Standard $3\times3$ convolution for local features.
  4. Frequency Path ($p=1$): Global processing in the Fourier domain.
  5. Fractional Path ($p=0.5$): Processing in a unified spatial-frequency domain.
  6. Fast Implementation: A major hurdle for FRFT has historically been computational cost. The authors developed a fast 2D implementation using matrix multiplication, making it as efficient as a standard Fast Fourier Transform (FFT) in practical settings.
The MFRFC architecture splits features into spatial, fractional, and frequency paths to capture diverse signal characteristics.

Key Insights and Ideas

  • Separability in the Fractional Domain: Signals that overlap and are inseparable in both the pure spatial and pure frequency domains often become clearly separable when viewed from a fractional angle. This allows the network to extract "cleaner" features that standard CNNs might miss.
  • Chirps vs. Sines: While the standard Fourier Transform uses sine waves as basis functions, the FRFT uses Linear Frequency Modulation (LFM) signals, also known as "chirps." These are mathematically much better suited for representing the non-stationary nature of images.
  • Unified Perspective: As the fractional order $p$ increases, the image smoothly transitions from clear spatial outlines to concentrated frequency energy. The intermediate orders contain a unique, rich mix of both structural and spectral information.
Amplitude and phase spectrums of FRFT at different orders. Spatial information gradually blurs into frequency energy as the order increases.

Key Results

The authors evaluated their MFRFC operator across an impressive breadth of benchmarks, including object detection (PASCAL VOC), image classification (CIFAR-10), denoising (SIDD), low-light enhancement, dehazing, deraining, and guided super-resolution.

The MFRFC operator consistently outperformed "Original" baselines and "SFC" (Spatial-Frequency Convolution) variants across all tested tasks:

  • Object Detection: Improved Faster RCNN's mAP from 77.26 to 77.96.
  • Low-light Enhancement: On the SID dataset, it boosted PSNR from 20.10 dB to 22.36 dB—a massive gain in image restoration quality.
  • Denoising: Achieved a higher PSNR (37.60) compared to the DnCNN baseline (37.19).
  • Efficiency: Crucially, the authors demonstrate that these performance gains come with a "negligible parameter increase," making MFRFC a highly viable drop-in replacement for standard convolutional layers.

Limitations and Future Work

While the paper presents a fixed order ($p=0.5$) for the fractional path to keep the architecture simple and efficient, the authors acknowledge that the optimal fractional order likely varies depending on the specific task or even the specific image content.

They briefly mention an extended version, MFRFC++, which allows the network to learn the optimal order adaptively. Future work could involve more deeply integrating these learnable rotations into the backbone of various architectures to see if "dynamic" spatial-frequency analysis provides even greater leaps in performance.

Summary updated

Abstract

Existing deep learning-based computer vision methods usually operate in the spatial and frequency domains, which are two orthogonal \textbf{individual} perspectives for image processing. In this paper, we introduce a new spatial-frequency analysis tool, Fractional Fourier Transform (FRFT), to provide comprehensive \textbf{unified} spatial-frequency perspectives. The FRFT is a unified continuous spatial-frequency transform that simultaneously reflects an image's spatial and frequency representations, making it optimal for processing non-stationary image signals. We explore the properties of the FRFT for image processing and present a fast implementation of the 2D FRFT, which facilitates its widespread use. Based on these explorations, we introduce a simple yet effective operator, Multi-order FRactional Fourier Convolution (MFRFC), which exhibits the remarkable merits of processing images from more perspectives in the spatial-frequency plane. Our proposed MFRFC is a general and basic operator that can be easily integrated into various tasks for performance improvement. We experimentally evaluate the MFRFC on various computer vision tasks, including object detection, image classification, guided super-resolution, denoising, dehazing, deraining, and low-light enhancement. Our proposed MFRFC consistently outperforms baseline methods by significant margins across all tasks.
BibTeX
@inproceedings{yu2023deep,
  author = {Hu Yu and Jie Huang and Lingzhi Li and Man Zhou and Feng Zhao},
  title = {{Deep Fractional Fourier Transform}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=YsYKv95jy9},
}
Figure