MSP-Former: Multi-Scale Projection Transformer for Single Image Desnowing

Sixiang Chen, Tian Ye, Yun Liu, Taodong Liao, Yi Ye, Erkang Chen

Abstract

Snow removal causes challenges due to its characteristic of complex degradations. To this end, targeted treatment of multi-scale snow degradations is critical for the network to learn effective snow removal. In order to handle the diverse scenes, we propose a multi-scale projection transformer (MSP-Former), which understands and covers a variety of snow degradation features in a multi-path manner, and integrates comprehensive scene context information for clean reconstruction via self-attention operation. For the local details of various snow degradations, the local capture module is introduced in parallel to assist in the rebuilding of a clean image. Such design achieves the SOTA performance on three desnowing benchmark datasets while costing the low parameters and computational complexity, providing a guarantee of practicality.

Introduction

Removing snow from a single image is a complex task involving overlapping degradations: distinct snow particles, the haze-like veiling effect, and chromatic aberration. Existing deep learning models often struggle to balance local texture preservation with global scene understanding. Furthermore, many high-performing models are computationally intensive, limiting their deployment in real-time or mobile applications.

MSP-Former addresses these challenges by introducing a lightweight, scale-aware transformer architecture. It combines a multi-scale projection mechanism for global context with a parallel Local Capture Module (LCB) for fine-grained details, achieving state-of-the-art (SOTA) performance with significantly reduced computational overhead.

Methodology

The MSP-Former utilizes a UNet-like encoder-decoder structure featuring dual-path blocks at each stage. The core innovation lies in the interaction between two parallel branches:

  1. Multi-Scale Projection (MSP) Module: This transformer-based branch employs multi-scale average pooling to aggregate features at various resolutions. These features are projected into Key (K) and Value (V) matrices, allowing the Query (Q) to attend to snow degradations of different sizes across the entire image. This pooling significantly reduces the computational complexity of the self-attention mechanism compared to standard transformers.
  2. Local Capture Module (LCB): Operating in parallel, the LCB uses depth-wise convolutions and channel attention to extract fine-grained local details and inter-channel relationships that might be missed by the global transformer.

A Channel Shuffle operation is used to integrate information between these branches before merging. The model is optimized using the robust Charbonnier Loss.

The MSP-Former architecture details the dual-path block, including the MSP Self-Attention and Local Capture Module (LCB).

Experiments and Results

The model was evaluated on three major benchmarks: the Comprehensive Snow Dataset (CSD), Snow Removal via Residual Subtraction (SRRS), and Snow 100K.

Performance and Efficiency

MSP-Former consistently outperformed previous SOTA methods like TransWeather and HDCW-Net. On the CSD dataset, it achieved a PSNR of 33.75 dB, surpassing the previous best of 31.76 dB. Remarkably, it achieves this with only 2.83 million parameters—nearly 8 times fewer than TransWeather (21.9M). In terms of computational complexity, it requires only 4.42 GMacs, making it significantly more efficient than competing high-end models.

Comparison of PSNR vs. GFLOPs across various desnowing models, highlighting the efficiency of MSP-Former.

Key Insights

  • Scale-Aware Attention: Multi-scale pooling allows the model to handle diverse snow sizes without a massive increase in parameters.
  • Efficiency through Pooling: Reducing the size of K and V matrices makes the self-attention operation highly efficient.
  • Parallel Processing: The dual-path design preserves local sharpness while maintaining a global understanding of the scene.

Conclusion

MSP-Former demonstrates that high-performance image restoration does not require massive computational resources. By intelligently combining scale-aware attention with local feature capture, it provides a practical solution for real-world desnowing. Future work may explore its performance in extreme blizzard conditions and its applicability to other adverse weather tasks like heavy rain or underwater enhancement.

Summary updated

Abstract

Snow removal causes challenges due to its characteristic of complex degradations. To this end, targeted treatment of multi-scale snow degradations is critical for the network to learn effective snow removal. In order to handle the diverse scenes, we propose a multi-scale projection transformer (MSP-Former), which understands and covers a variety of snow degradation features in a multi-path manner, and integrates comprehensive scene context information for clean reconstruction via self-attention operation. For the local details of various snow degradations, the local capture module is introduced in parallel to assist in the rebuilding of a clean image. Such design achieves the SOTA performance on three desnowing benchmark datasets while costing the low parameters and computational complexity, providing a guarantee of practicality.
BibTeX
@inproceedings{chen2023msp,
  author = {Sixiang Chen and Tian Ye and Yun Liu and Taodong Liao and Yi Ye and Erkang Chen},
  title = {{MSP-Former: Multi-Scale Projection Transformer for Single Image Desnowing}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095605},
}
Figure