TL;DR
This paper introduces F2former, a novel image deblurring method that leverages the Fractional Fourier Transform (FRFT) to better handle non-uniform blurs. By integrating FRFT into both a deconvolution layer and a transformer architecture, F2former achieves state-of-the-art performance on various deblurring tasks, outperforming existing methods that struggle with spatially varying blur.
What Problem Does This Paper Address?
Image deblurring aims to recover a sharp image from a blurry input, often modeled as a convolution of the clean image with a blur kernel, plus noise. While traditional methods focused on estimating the blur kernel, modern deep learning approaches often operate directly in the spatial or frequency domains. However, standard Fourier Transform (FT) methods are limited in their ability to capture spatially varying blur patterns because FT assumes signal stationarity. This limitation hinders performance in real-world scenarios where blur is often non-uniform. Existing deep learning methods, even those using transformers, can also struggle with representing and processing these complex blur characteristics effectively.
Methods
The core innovation of F2former lies in its extensive use of the Fractional Fourier Transform (FRFT). FRFT is a generalization of FT that provides a unified spatial-frequency representation, making it more adept at handling non-stationary signals like images with varying blur.
The F2former architecture is a U-Net-like encoder-decoder model with several key components:
- Fractional Feature-based Wiener Deconvolution (F2WD): This layer operates at the shallow feature level. It uses FRFT to perform Wiener deconvolution in the feature space, aiming to refine low-level features and produce sharper representations before they enter the main transformer blocks. This is inspired by previous work on feature-based Wiener deconvolution but adapted for the FRFT domain.
- Fractional Hybrid Transformer Block (FHTB): This is the main building block for image reconstruction. Each FHTB consists of:
- Fractional Fourier based Feature Refinement module (F3RB): This module uses FRFT to extract features at different context levels.
- Fractional Frequency aware Transformer Block (F2TB): This block is crucial for capturing spatio-spectral information. It comprises two sub-modules:
- Fractional Frequency aware Self-Attention (F2SA): This module computes self-attention in the FRFT domain. It uses a phase-modulated self-attention mechanism to emphasize important frequency components and has a computational complexity similar to FFT-based methods (O(N log N)), making it efficient.
- Frequency Division Multiplexing-based FFN (FM-FFN): This module dynamically extracts high and low-frequency features. It uses FT (where FRFT with $\alpha=1$ converges to FT) to split features into frequency bands and then selectively processes them using a cosine bell function for smooth transitions and to avoid ringing artifacts.
The FRFT order ($\alpha$) is treated as a learnable parameter in F2WD and F2SA.
Datasets
The paper evaluates F2former on several standard benchmark datasets for image deblurring:
- GoPro: Used for motion deblurring.
- HIDE: Used for motion deblurring.
- RealBlur-R and RealBlur-J: Datasets for real-world motion deblurring.
- Dual-Pixel Defocus Deblurring (DPDD): Used for defocus deblurring, containing both indoor and outdoor scenes.
Key Insights and Ideas
- FRFT for Non-Stationary Blur: The central idea is that FRFT's ability to represent signals in a unified spatial-frequency domain makes it inherently better suited for handling non-uniform and spatially varying blurs compared to traditional FT.
- Feature-Level Deblurring with FRFT: Applying Wiener deconvolution in the FRFT domain at the feature level (F2WD) effectively sharpens features, which is crucial for subsequent reconstruction stages.
- Spatio-Spectral Attention: F2SA leverages FRFT to perform self-attention not just in the spatial domain but also considering frequency components, allowing it to capture more relevant information for deblurring.
- Frequency-Selective Feature Processing: FM-FFN's ability to explicitly separate and process high and low-frequency components allows for more targeted feature refinement, which is vital for restoring details lost in blur.
- Efficiency of FRFT-based Attention: The F2SA module achieves efficiency comparable to FFT-based attention mechanisms, addressing a common bottleneck in transformer models.
Key Results
F2former demonstrates state-of-the-art performance across multiple benchmark datasets:
- GoPro: Achieves a gain of 0.53 dB in PSNR over the second-best method for the base model.
- HIDE and RealBlur-J: Gains of over 1 dB in PSNR are reported.
- DPDD: F2former achieves significant performance gains, with more than 2 dB improvement for both single and dual-pixel defocus deblurring tasks.
- RealBlur: F2former shows the best performance with a gain of 0.5-0.6 dB in PSNR over the second-best method.
- Efficiency: While FRFT implementations can be slower than optimized FFT, F2former's runtime is competitive. The model also offers parameter efficiency, with the small version having 50% fewer parameters than the base model, albeit with a slight performance drop.
Qualitative results (visualizations) show F2former effectively removing non-uniform blurs and producing sharper images compared to other methods.
Limitations and Future Work
- Kernel Estimation Dependency: F2former relies on a separate, pre-trained kernel estimation block (KEB). The authors acknowledge that this dependency can be a limitation.
- CNN Linearity Assumption: The F2WD layer assumes CNN operations are locally linear, which might occasionally lead to erroneous feature maps.
- Future Work: The authors propose investigating blur kernel estimation directly in the feature space and integrating it into an end-to-end training framework. They also plan to extend FRFT's application to other image restoration tasks like super-resolution, denoising, and dehazing.
Summary updated