Hybrid Frequency Modulation Network for Image Restoration

Yuning Cui, Mingyu Liu, Wenqi Ren, Alois Knoll

Abstract

Image restoration involves recovering a high-quality image from its corrupted counterpart. This paper presents an effective and efficient framework for image restoration, termed CSNet, based on "channel + spatial" hybrid frequency modulation. Different feature channels include different degradation patterns and degrees; however, most current networks ignore the importance of channel interactions. To alleviate this issue, we propose a frequency-based channel feature modulation module to facilitate channel interactions through the channel-dimension Fourier transform. Furthermore, based on our observations, we develop a multi-scale frequency-based spatial feature modulation module to refine the direct-current component of features using extremely lightweight learnable parameters. This module contains a densely connected coarse-to-fine learning paradigm for enhancing multi-scale representation learning. In addition, we introduce a frequency-inspired loss function to achieve omni-frequency learning. Extensive experiments on nine datasets demonstrate that the proposed network achieves state-of-the-art performance for three image restoration tasks, including image dehazing, image defocus deblurring, and image desnowing.

Introduction

Image restoration—removing haze, snow, or blur—is a classic "ill-posed" problem in computer vision. While Convolutional Neural Networks (CNNs) have been the standard tool, their local receptive fields often struggle to capture global context, such as the uniform veil of haze across an entire landscape. Transformers were introduced to address this via self-attention mechanisms, but they are notoriously computationally expensive and memory-heavy.

Existing frequency-domain networks attempt to bridge this gap by using the Fast Fourier Transform (FFT) to gain a global receptive field. However, most only apply FFT to spatial dimensions ($H \times W$), ignoring the fact that different feature channels often contain distinct degradation patterns. CSNet (Hybrid Frequency Modulation Network) addresses this by modulating both channel and spatial frequencies to capture global degradation patterns more effectively.

Methodology

The CSNet architecture follows a U-shaped encoder-decoder structure, incorporating three key innovations:

  1. Frequency-based Channel Feature Modulation (FCFM): Unlike standard channel attention, which assigns a single weight to a channel, FCFM applies an FFT across the channel dimension. This allows each pixel to perceive signals and interactions across all channels in multiple spectral spaces, improving the handling of spatially varying degradations.
  2. Multi-scale Frequency-based Spatial Feature Modulation (MFSFM): The authors observed that the "Direct Current" (DC) component—the lowest frequency in a signal—is the primary carrier of global degradation information. MFSFM targets and refines this DC component using a coarse-to-fine, densely connected learning paradigm. It utilizes Global Average Pooling (GAP) to extract the DC part efficiently without requiring a full spatial FFT.
  3. Frequency-inspired Loss Function (FLF): To ensure the model restores sharp edges and fine details, this loss function specifically penalizes discrepancies in the high-frequency domain, forcing the model to achieve omni-frequency learning.

Experimental Results

CSNet was evaluated on nine datasets across three tasks: image dehazing (SOTS, Dense-Haze, NH-HAZE, O-HAZE, NHR, and GTA5), defocus deblurring (DPDD), and image desnowing (CSD and Snow100K).

Quantitative Performance and Efficiency

CSNet consistently outperformed existing state-of-the-art (SOTA) models. On the SOTS dehazing dataset, it achieved a PSNR of 41.34 dB, surpassing the strong MB-TaylorFormer-B by 0.63 dB. Notably, it maintains a significantly lower computational footprint than heavy Transformer-based models, as shown in the efficiency comparison below.

Comparison of PSNR vs. FLOPs for various image restoration models. CSNet achieves superior performance with lower computational complexity.

Qualitative Analysis

Visual results demonstrate that CSNet is particularly effective at removing lingering haze and "hard" blurs that other models often leave behind as artifacts. The model's ability to refine the DC component allows it to handle global degradation patterns that local-only methods struggle with.

Qualitative comparison of hazy images, CSNet restoration results, and ground truth (GT) images with corresponding PSNR values.

Conclusion

CSNet introduces an efficient "channel + spatial" hybrid frequency modulation strategy that captures global degradation patterns effectively. By focusing on channel-wise interactions and DC component refinement, it achieves SOTA performance across multiple restoration tasks while remaining computationally efficient. Future work may explore the integration of these frequency modules into different backbones and their generalization to other degradation types like rain or motion blur.

Summary updated

Abstract

Image restoration involves recovering a high-quality image from its corrupted counterpart. This paper presents an effective and efficient framework for image restoration, termed CSNet, based on ``channel + spatial" hybrid frequency modulation. Different feature channels include different degradation patterns and degrees, however, most current networks ignore the importance of channel interactions. To alleviate this issue, we propose a frequency-based channel feature modulation module to facilitate channel interactions through the channel-dimension Fourier transform. Furthermore, based on our observations, we develop a multi-scale frequency-based spatial feature modulation module to refine the direct-current component of features using extremely lightweight learnable parameters. This module contains a densely connected coarse-to-fine learning paradigm for enhancing multi-scale representation learning. In addition, we introduce a frequency-inspired loss function to achieve omni-frequency learning. Extensive experiments on nine datasets demonstrate that the proposed network achieves state-of-the-art performance for three image restoration tasks, including image dehazing, image defocus deblurring, and image desnowing. The code and models are available at https://github.com/c-yn/CSNet.
BibTeX
@inproceedings{cui2024hybrid,
  author = {Yuning Cui and Mingyu Liu and Wenqi Ren and Alois Knoll},
  title = {{Hybrid Frequency Modulation Network for Image Restoration}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=WsspD1QbZQ},
}
Figure