TL;DR
This paper introduces "Weamba," a novel vision state space model for restoring remote sensing images degraded by weather conditions like haze and rain. By combining a local-enhanced state space module with a unique multi-router scanning strategy, Weamba effectively captures both local and global image features with linear complexity, outperforming existing CNN and Transformer-based methods.
What Problem Does This Paper Address?
Remote sensing (RS) images are crucial for various applications, but adverse weather conditions such as haze and rain significantly degrade their quality, hindering accurate analysis. While recent deep learning methods like CNNs and Transformers have improved image restoration, they face limitations: CNNs have restricted receptive fields, and Transformers incur quadratic computational costs, especially for high-resolution RS images. This creates a need for efficient models that can capture both local details and long-range dependencies in degraded RS imagery.
Methods
The core of Weamba is a local-enhanced state space module (LSSM), which integrates two key components:
- Global State Space Block (GSSB): This block is inspired by the Transformer and Mamba architectures and is designed for global feature extraction. It incorporates a novel Multi-Router Scanning Block (MRSB).
- Local Feature Modulation Block (LFMB): This block works in parallel with the GSSB to extract local features, complementing the global information.
The Multi-Router Scanning Block (MRSB) is a key innovation. Unlike previous methods that use single scanning directions (e.g., row-wise, column-wise, or specific shapes like 'Z'), MRSB employs three distinct routing pathways ('Z', 'S', and 'W' shapes) to scan image patches. This multi-route approach aims to capture spatially varying information more comprehensively and reduce redundancy caused by repeated scanning directions.
The overall architecture follows an encoder-decoder structure, with LSSMs forming the backbone of each level. Skip connections are used to preserve information across levels. The model is trained using an L1 loss function.
Datasets
The paper evaluates Weamba on several publicly available datasets for both haze and rain removal:
- Haze Datasets:
- SateHazelk (3 different haze densities)
- RS-Haze
- RICE
- RRSD300 (real-world hazy images)
- Rain Datasets:
- UAV-Rain1k
Key Insights and Ideas
- Bridging Local and Global: The paper highlights that effective RS image restoration requires capturing both fine local details and broad global context. Weamba's LSSM, by combining GSSB and LFMB, explicitly addresses this.
- The Problem with Single-Path Scanning: The authors argue that existing methods using single scanning directions (even if multi-directional) can lead to redundant information and limited performance gains.
- Multi-Router Scanning for Richer Features: The proposed MRSB with its 'Z', 'S', and 'W' shaped scanning pathways is shown to be more effective at capturing diverse spatial information, leading to better feature representation.
- State Space Models for Efficiency: By leveraging state space models (specifically Mamba's principles), Weamba achieves linear complexity, making it more efficient for high-resolution RS images compared to quadratic-complexity Transformers.
Key Results
- Quantitative Performance: Weamba consistently outperforms state-of-the-art CNN, Transformer, and Mamba-based methods on various benchmark datasets for both haze and rain removal. For instance, it achieves the highest PSNR on the UAV-Rain1k dataset and shows significant improvements over methods like RSDformer.
- Efficiency: Weamba exhibits a markedly lower number of parameters and FLOPs compared to many existing methods, making it computationally efficient. It also achieves a faster inference speed.
- Qualitative Results: Visual comparisons demonstrate that Weamba recovers clearer images with preserved intricate details and more accurate color representation, effectively removing haze and rain artifacts.
- Application to Object Detection: Restored images from Weamba lead to improved object detection accuracy, particularly in identifying structures like houses, highlighting the practical utility of the method.
Limitations and Future Work
- Linear Scanning Limitations: The authors acknowledge that even their multi-router scanning strategy, while improved, still relies on linear scanning curves (like Zigzag). They suggest that these might inherently limit the ability to fully capture complex spatial relationships.
- Computational Complexity of MRSB: While Weamba is efficient overall, the multi-router scanning strategy, despite its benefits, could potentially introduce more computational complexity.
- Future Directions:
- Explore fractal scanning curves or dynamic adaptive scanning for more advanced patch serialization.
- Further investigate the balance between computational complexity and processing capability.
Summary updated