A Comparative Study of Image Restoration Networks for General Backbone Network Design

Xiangyu Chen, Zheyuan Li, Yuandong Pu, Yihao Liu, Jiantao Zhou, Yu Qiao, Chao Dong

Abstract

Image restoration encompasses a variety of degradations, from noise and blur to low resolution. While specialized architectures often excel at specific tasks, they frequently fail to generalize across others. This paper presents X-Restormer, a versatile backbone that bridges the gap between local spatial mapping and global context interaction. By integrating hybrid attention mechanisms into a multi-scale U-shaped architecture, X-Restormer establishes a new state-of-the-art for general-purpose image restoration.

Introduction: The Specialist Trap

In computer vision, image restoration is an umbrella term for distinct mathematical problems: denoising, deblurring, super-resolution (SR), deraining, and dehazing. Historically, research has favored "specialist" networks. For instance, SwinIR dominates SR by focusing on fine local textures but falters in deblurring due to a lack of global receptive fields. Conversely, Restormer excels at global tasks like dehazing but produces suboptimal results in SR. The core challenge is task generality: creating a single backbone capable of handling diverse degradation types without extensive per-task redesign.

Methodology: Hybridizing Attention for Generality

To address the trade-off between local spatial mapping and global information interaction, the authors developed X-Restormer. The architecture is built upon a 4-level encoder-decoder (U-Shape) foundation, which is critical for multi-scale processing of large-scale degradations like motion blur.

Comparison of core operators: Convolution, Spatial self-attention, and Transposed self-attention.

The primary innovation lies in the Hybrid Attention Blocks. Instead of relying on a single attention mechanism, X-Restormer alternates between two complementary types:

  1. Transposed Self-Attention (TSA): Borrowed from Restormer, this operates on the channel dimension to capture global context with low computational cost.
  2. Spatial Self-Attention (SSA): Specifically, the authors use Overlapping Cross-Attention (OCA). This focuses on local pixel relationships, providing the "spatial mapping" power that previous global-focused models lacked.

This hybrid approach allows the model to satisfy the functional requirements of both local detail reconstruction and global context understanding without increasing the parameter count (remaining at approximately 26M).

Experimental Evaluation

The models were evaluated on standard benchmarks including DF2K for SR, DFWB for denoising, GoPro for deblurring, Rain13K for deraining, and RESIDE for dehazing.

Key Results

X-Restormer achieved state-of-the-art (SOTA) or near-SOTA performance across all five categories:

  • Super-Resolution: Outperformed Restormer by 0.34dB on the challenging Urban100 dataset.
  • Dehazing: Achieved a significant 0.93dB improvement over Restormer on the SOTS Indoor benchmark.
  • Deblurring: Gained 0.36dB over NAFNet on the GoPro dataset.
Relative performance of X-Restormer compared to other backbones across five restoration tasks.

In "All-in-One" training scenarios—where a single model is trained simultaneously on all tasks—X-Restormer’s balanced design naturally mitigates "optimization conflicts" that typically degrade the performance of more specialized architectures like SwinIR.

Discussion and Insights

The failure of previous models like Restormer in SR tasks was traced back to a reliance on depth-wise convolutions for spatial information. While efficient, these lack the power of window-based self-attention for reconstructing fine textures. X-Restormer demonstrates that a general backbone does not necessarily require more parameters, but rather a more comprehensive set of functional operators.

Limitations and Future Work

Despite its success as a generalist, X-Restormer does not yet surpass the absolute best "pure" SR specialists (e.g., HAT). The U-shape architecture, while beneficial for blur, can lead to the loss of some high-frequency details during downsampling. Future research will focus on preserving high-frequency information within multi-scale structures and further optimizing the computational efficiency of spatial attention to reduce the slight increase in FLOPs compared to the original Restormer.

Summary updated

BibTeX
@inproceedings{chen2024comparative,
  author = {Xiangyu Chen and Zheyuan Li and Yuandong Pu and Yihao Liu and Jiantao Zhou and Yu Qiao and Chao Dong},
  title = {{A Comparative Study of Image Restoration Networks for General Backbone Network Design}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73209-6\_5},
}
Figure