TL;DR
This paper introduces a novel multi-input, multi-output Transformer architecture (MSDT) for image deraining that leverages multi-scale representations. By processing images at different resolutions and intelligently fusing their features, MSDT significantly improves deraining performance, outperforming existing state-of-the-art methods.
What Problem Does This Paper Address?
Computer vision systems struggle with adverse weather conditions, particularly rainy days, which degrade image quality and hinder performance. Single image deraining aims to remove rain streaks and restore clear images. While deep learning, especially Transformers, has shown great promise, existing methods primarily rely on a single-input, single-output U-Net architecture. This approach overlooks valuable information present across different image scales – coarser scales often reveal less intense rain, making it easier to discern underlying clean image details. Furthermore, these methods fail to exploit the complementary information that exists between different scales. This paper addresses this limitation by proposing a framework that explicitly models and fuses multi-scale information for more robust and effective image deraining.
Methods
The core of the proposed solution is the Multi-Scale Deraining Transformer (MSDT), a multi-input, multi-output framework. Instead of a single input, MSDT takes a pyramid of rainy images (original, 1/2 scale, 1/4 scale) as input. The network is structured into three scales (S1, S2, S3), corresponding to the finest to coarsest resolutions.
Key components include:
- Coupled Representation Module (CRM): This module is designed to jointly learn intra-scale features. It comprises two sub-blocks:
- Frequency-Enhanced Convolutional Block (FECB): This block uses Fast Fourier Transform (FFT) to extract high-frequency components, aiming to recover texture details. It then applies convolutional layers and an inverse FFT to reconstruct the spatial structure.
- Sparsity-Compensated Transformer Block (SCTB): This block enhances non-local information by introducing a "neighbor softmax" mechanism. Unlike standard Transformers that consider all tokens, this approach focuses attention on spatially proximate tokens, reducing redundancy and improving feature aggregation. Different thresholds are used for different scales to adapt to varying rain intensities.
- Gated Fusion Module (GFM): This module facilitates inter-scale communication by adaptively aggregating features from different scales. It uses dual gate units to dynamically control hierarchical information flow, ensuring that scale-specific representations are optimally integrated.
The MSDT employs a coarse-to-fine deraining strategy, starting restoration from the coarsest scale and progressively refining it to the finest scale. This approach is hypothesized to simplify the learning process by providing initial solutions from coarser scales.
Datasets
The paper evaluates MSDT on five publicly available datasets:
- Rain200L and Rain200H: These datasets contain synthetic rainy images.
- DID-Data and DDN-Data: These datasets feature synthetic rainy images with distinct rain directions and densities.
- SPA-Data: This is a large-scale real-world rain benchmark.
Key Insights and Ideas
- Coarser scales are treasure troves for clean image signals: The observation that rain intensity significantly reduces at coarser scales is crucial. This allows underlying clean image features to "naturally emerge," providing valuable clues for deraining that are often obscured at the finest scale.
- Multi-scale processing isn't just about resolution, but complementary information: The paper highlights that different scales offer distinct types of information – global context at coarser scales and local details at finer scales. Effectively fusing these is key to robust deraining.
- Sparsity in attention is beneficial: The "neighbor softmax" in SCTB demonstrates that focusing attention on relevant neighbors, rather than all tokens, can lead to more focused and robust feature representations, especially in the context of image deraining.
- Coarse-to-fine deraining simplifies learning: By starting deraining from coarser scales, the model gets a "head start," potentially reducing the difficulty of learning complex mappings for fine-grained details.
Key Results
MSDT achieves state-of-the-art performance on five benchmarks, consistently outperforming existing methods.
- It surpasses the recent Transformer-based approach DRSformer by 0.52 dB in PSNR on the Rain200L benchmark.
- Qualitative results show MSDT producing cleaner images with better detail restoration compared to other methods, even on real-world data.
- In terms of model efficiency, MSDT achieves competitive results in PSNR, SSIM, parameters, and inference time, positioning it as a strong contender among Transformer-based deraining models.
Limitations and Future Work
- Computational Complexity: While competitive, MSDT's multi-input nature and Transformer components contribute to its computational cost. The paper notes that it is not the most efficient in terms of FLOPs compared to some other methods, suggesting room for optimization.
- Future Directions: The authors suggest that their multi-input, multi-output architecture could be a valuable framework for other low-level vision tasks that can benefit from multi-scale feature analysis. Further exploration into optimizing the balance between performance and efficiency is also implied.
Summary updated