Abstract
Image restoration involves recovering a high-quality image from its corrupted counterpart. This paper presents an effective and efficient framework for image restoration, termed CSNet, based on "channel + spatial" hybrid frequency modulation. Different feature channels include different degradation patterns and degrees; however, most current networks ignore the importance of channel interactions. To alleviate this issue, we propose a frequency-based channel feature modulation module to facilitate channel interactions through the channel-dimension Fourier transform. Furthermore, based on our observations, we develop a multi-scale frequency-based spatial feature modulation module to refine the direct-current component of features using extremely lightweight learnable parameters. This module contains a densely connected coarse-to-fine learning paradigm for enhancing multi-scale representation learning. In addition, we introduce a frequency-inspired loss function to achieve omni-frequency learning. Extensive experiments on nine datasets demonstrate that the proposed network achieves state-of-the-art performance for three image restoration tasks, including image dehazing, image defocus deblurring, and image desnowing.
Introduction
Image restoration—removing haze, snow, or blur—is a classic "ill-posed" problem in computer vision. While Convolutional Neural Networks (CNNs) have been the standard tool, their local receptive fields often struggle to capture global context, such as the uniform veil of haze across an entire landscape. Transformers were introduced to address this via self-attention mechanisms, but they are notoriously computationally expensive and memory-heavy.
Existing frequency-domain networks attempt to bridge this gap by using the Fast Fourier Transform (FFT) to gain a global receptive field. However, most only apply FFT to spatial dimensions ($H \times W$), ignoring the fact that different feature channels often contain distinct degradation patterns. CSNet (Hybrid Frequency Modulation Network) addresses this by modulating both channel and spatial frequencies to capture global degradation patterns more effectively.
Methodology
The CSNet architecture follows a U-shaped encoder-decoder structure, incorporating three key innovations:
- Frequency-based Channel Feature Modulation (FCFM): Unlike standard channel attention, which assigns a single weight to a channel, FCFM applies an FFT across the channel dimension. This allows each pixel to perceive signals and interactions across all channels in multiple spectral spaces, improving the handling of spatially varying degradations.
- Multi-scale Frequency-based Spatial Feature Modulation (MFSFM): The authors observed that the "Direct Current" (DC) component—the lowest frequency in a signal—is the primary carrier of global degradation information. MFSFM targets and refines this DC component using a coarse-to-fine, densely connected learning paradigm. It utilizes Global Average Pooling (GAP) to extract the DC part efficiently without requiring a full spatial FFT.
- Frequency-inspired Loss Function (FLF): To ensure the model restores sharp edges and fine details, this loss function specifically penalizes discrepancies in the high-frequency domain, forcing the model to achieve omni-frequency learning.
Experimental Results
CSNet was evaluated on nine datasets across three tasks: image dehazing (SOTS, Dense-Haze, NH-HAZE, O-HAZE, NHR, and GTA5), defocus deblurring (DPDD), and image desnowing (CSD and Snow100K).
Quantitative Performance and Efficiency
CSNet consistently outperformed existing state-of-the-art (SOTA) models. On the SOTS dehazing dataset, it achieved a PSNR of 41.34 dB, surpassing the strong MB-TaylorFormer-B by 0.63 dB. Notably, it maintains a significantly lower computational footprint than heavy Transformer-based models, as shown in the efficiency comparison below.
Qualitative Analysis
Visual results demonstrate that CSNet is particularly effective at removing lingering haze and "hard" blurs that other models often leave behind as artifacts. The model's ability to refine the DC component allows it to handle global degradation patterns that local-only methods struggle with.
Conclusion
CSNet introduces an efficient "channel + spatial" hybrid frequency modulation strategy that captures global degradation patterns effectively. By focusing on channel-wise interactions and DC component refinement, it achieves SOTA performance across multiple restoration tasks while remaining computationally efficient. Future work may explore the integration of these frequency modules into different backbones and their generalization to other degradation types like rain or motion blur.
Summary updated