Exploring Real&Synthetic Dataset and Linear Attention in Image Restoration

Yuzhen Du, Teng Hu, Jiangning Zhang, Ran Yi, Chengming Xu, Xiaobin Hu, Kai Wu, Donghao Luo, Yabiao Wang, Lizhuang MaarXiv 2024 · DOI · OpenReview

TL;DR

This paper introduces "ReSyn," a new, large-scale dataset for image restoration that addresses the domain gap caused by imbalanced image complexity in existing datasets. They also propose RWKV-IR, a novel transformer-based image restoration model that leverages linear attention mechanisms for improved efficiency and performance, particularly by incorporating a new "Depth-wise Convolution Shift" and "Cross-Bi-WKV" module.

What Problem Does This Paper Address?

Image restoration (IR) tasks, like denoising or super-resolution, aim to recover high-quality images from degraded versions. While deep learning has made huge strides, current IR research faces a few key challenges:

  1. Lack of Standardized Benchmarks: There's no unified way to train and evaluate IR models, making it hard to compare different approaches fairly. This includes inconsistent training iterations and configurations.
  2. Dataset Domain Gap: Existing training and testing datasets often have different distributions of "image complexity" (how detailed or textured an image is). This mismatch can lead to models that perform well on training data but poorly on real-world test data.
  3. Computational Cost of Transformers: While powerful, transformer models for IR can be computationally expensive due to their quadratic attention mechanisms.

Methods

The authors tackle these problems with several novel contributions:

  • Image Complexity Metric: They propose using the Gray-Level Co-occurrence Matrix (GLCM) to quantify image complexity. This metric is shown to be a better predictor of restoration performance than traditional metrics like Bits Per Pixel (BPP).
  • ReSyn Dataset: Using the GLCM metric, they construct a new, large-scale dataset called ReSyn. This dataset balances image complexity by filtering both real and AI-generated (AIGC) images, aiming to bridge the domain gap.
  • Unified Training Standard: They establish a standardized benchmark for IR training, specifying training iterations and configurations to ensure fair model comparison, focusing on both convergence speed and restoration capability.
  • RWKV-IR Model: They introduce RWKV-IR, a transformer-based IR model that incorporates the RWKV (Receptance Weighted Key-Value) linear attention mechanism. This allows for global receptive fields with linear computational complexity.
  • Depth-wise Convolution Shift (DC-Shift): To better capture local image features, they replace RWKV's original "Q-shift" with a novel DC-Shift module, which uses depth-wise convolutions.
  • Cross-Bi-WKV Module: To address the unbalanced attention in RWKV's original bidirectional attention (Bi-WKV), they propose a Cross-Bi-WKV module that combines two Bi-WKV modules with different scanning orders (horizontal and vertical) for more balanced attention.

Datasets

  • Training/Evaluation Datasets: DIV2K, DF2K, Urban100, Manga109, Set5, Set14, BSDS100, Kodak24, McMaster, DFWB RGB, DFWB-gray, Classic5, LIVE1.
  • New Benchmark Dataset: ReSyn. This dataset is unique in that it balances image complexity using a GLCM-based filtering method and includes both real and AIGC synthetic images.

Key Insights and Ideas

  • Image Complexity Matters: The paper strongly suggests that image complexity, as measured by GLCM, is a critical factor in dataset design for IR. Simply focusing on resolution or BPP is insufficient.
  • Linear Attention for IR: RWKV's linear attention mechanism offers a promising avenue for efficient and effective image restoration, overcoming the quadratic complexity of standard transformers.
  • Local vs. Global Attention: The authors highlight that standard transformer attention mechanisms, while good at global context, can struggle with local details crucial for IR. Their DC-Shift and Cross-Bi-WKV modules are designed to bridge this gap.
  • AIGC Integration: The inclusion of AIGC synthetic images in ReSyn is forward-thinking, acknowledging the growing role of synthetic data in AI research.

Key Results

  • RWKV-IR Performance: The proposed RWKV-IR model achieves state-of-the-art results on various image super-resolution tasks, often outperforming existing methods like SwinIR and MambaIR, especially with fewer training iterations. For example, it shows a PSNR improvement of 0.87dB over the Q-shift baseline and 0.75dB over the original Bi-WKV.
  • ReSyn Dataset Effectiveness: Training on the ReSyn dataset, even with its diverse sources, yields competitive performance for lightweight models, demonstrating the dataset's quality and the effectiveness of the complexity-balancing approach.
  • Benchmark Study: The paper presents comprehensive benchmark results comparing RWKV-IR and other SOTA methods across multiple datasets and training regimes, showcasing RWKV-IR's strong convergence and restoration capabilities.

Limitations and Future Work

  • RWKV Iteration: The authors acknowledge that RWKV is an evolving architecture, and future versions are expected to bring further improvements in global attention capabilities.
  • Ablation Study Focus: While the ablation studies are thorough, they primarily focus on the Urban100 dataset for lightweight models, suggesting potential for broader validation.
  • Generalizability: While the paper demonstrates effectiveness across various IR tasks (SR, denoising, JPEG artifact reduction), further exploration of RWKV-IR's adaptability to more diverse and challenging IR scenarios could be beneficial.

Summary updated

Abstract

Image restoration (IR) aims to recover high-quality images from degraded inputs, with recent deep learning advancements significantly enhancing performance. However, existing methods lack a unified training benchmark for iterations and configurations. We also identify a bias in image complexity distributions between commonly used IR training and testing datasets, resulting in suboptimal restoration outcomes. To address this, we introduce a large-scale IR dataset called ReSyn, which employs a novel image filtering method based on image complexity to ensure a balanced distribution and includes both real and AIGC synthetic images. We establish a unified training standard that specifies iterations and configurations for image restoration models, focusing on measuring model convergence and restoration capability. Additionally, we enhance transformer-based image restoration models using linear attention mechanisms by proposing RWKV-IR, which integrates linear complexity RWKV into the transformer structure, allowing for both global and local receptive fields. Instead of directly using Vision-RWKV, we replace the original Q-Shift in RWKV with a Depth-wise Convolution shift to better model local dependencies, combined with Bi-directional attention for comprehensive linear attention. We also introduce a Cross-Bi-WKV module that merges two Bi-WKV modules with different scanning orders for balanced horizontal and vertical attention. Extensive experiments validate the effectiveness of our RWKV-IR model.
BibTeX
@misc{du2024exploring,
  author = {Yuzhen Du and Teng Hu and Jiangning Zhang and Ran Yi and Chengming Xu and Xiaobin Hu and Kai Wu and Donghao Luo and Yabiao Wang and Lizhuang Ma},
  title = {{Exploring Real\&Synthetic Dataset and Linear Attention in Image Restoration}},
  year = {2024},
  eprint = {2412.03814},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.03814},
}
Figure