MSP-Former: Multi-Scale Projection Transformer for Single Image Desnowing

Tian Ye, Sixiang Chen, Tian Ye 0001, Yun Liu, Taodong Liao, Jingxia Jiang, Erkang Chen, Peng Chen

TL;DR

MSP-Former is a lightweight, high-performance transformer architecture designed to strip snow from single images. By combining a multi-scale projection mechanism for global context with a parallel "Local Capture Module" for fine details, it achieves state-of-the-art (SOTA) results while using significantly fewer parameters and less computational power than previous models.

What Problem Does This Paper Address?

Removing snow from a single image is a "multi-headed beast" of a problem. Unlike simple noise, snow degradation consists of several overlapping effects:

  1. Snow particles: Distinct flakes or streaks that occlude the background.
  2. Veiling effect: A haze-like blur caused by light scattering.
  3. Chromatic aberration: Color distortions around snow particles.

Existing deep learning models often struggle because they either focus too much on local textures (missing the "big picture" of the scene) or use standard transformers that treat all image patches the same way, regardless of the snow's scale. Furthermore, many high-performing models are computationally "heavy," making them impractical for real-time use or deployment on mobile devices. The authors of MSP-Former set out to create a model that is both scale-aware and highly efficient.

Methods

The MSP-Former uses a UNet-like encoder-decoder structure, but the "secret sauce" lies in its dual-path blocks. At each stage, the features are split into two parallel branches:

  • Multi-Scale Projection (MSP) Module: This is the transformer branch. Instead of standard self-attention, it uses multi-scale average pooling to aggregate features at different resolutions. These aggregated features are projected into the Key (K) and Value (V) matrices. This allows the Query (Q) to attend to snow degradations of various sizes (from tiny specks to large flakes) across the entire image.
  • Local Capture Module (LCB): Running in parallel to the transformer, this branch uses depth-wise convolutions and channel attention. It is designed to "excavate" fine-grained local details and inter-channel relationships that the global transformer might overlook.

To ensure these two branches talk to each other, the authors use a Channel Shuffle operation, which reorganizes information across the channel dimension before the features are merged back together. The entire system is trained using the Charbonnier Loss, a robust reconstruction loss function.

Datasets

The authors evaluated MSP-Former on three widely recognized desnowing benchmarks:

  • CSD (Comprehensive Snow Dataset): A large-scale synthetic dataset.
  • SRRS (Snow Removal via Residual Subtraction): Focuses on the relationship between snow and haze-like veiling.
  • Snow 100K: A massive dataset containing 100,000 synthetic snowy images.
  • Real-world images: Used for qualitative (visual) testing to ensure the model generalizes beyond synthetic data.

Key Insights and Ideas

  • Scale-Aware Attention: The "aha moment" here is that you don't need a massive transformer to handle complex snow. By simply pooling the Key and Value matrices at different scales, the model becomes inherently aware of different snow sizes without a massive increase in math.
  • Efficiency through Pooling: By using average pooling to reduce the size of the K and V matrices, the authors drastically reduced the computational complexity of the self-attention mechanism, which is usually the most "expensive" part of a transformer.
  • The Power of Parallelism: Instead of stacking CNN layers and Transformer layers sequentially, running them in parallel (MSP + LCB) allows the model to preserve local sharpness while simultaneously understanding global scene context.

Key Results

The MSP-Former outperformed all previous SOTA methods (including TransWeather and HDCW-Net) across all metrics:

  • Accuracy: On the CSD dataset, it achieved a PSNR of 33.75 dB, a significant jump over the previous best of 31.76 dB.
  • Lightweight Design: It uses only 2.83 million parameters. For context, the previous leader, TransWeather, uses 21.9 million—making MSP-Former nearly 8 times smaller.
  • Speed/Complexity: It requires only 4.42 GMacs of computation, making it more efficient than almost all competing high-end models.
  • Visual Quality: Qualitative results show the model is particularly good at removing small, stubborn snow spots and restoring the original color regions that other models often leave "washed out."

Limitations and Future Work

While the paper demonstrates excellent results, a few areas remain for exploration:

  • Extreme Conditions: The paper primarily tests on synthetic datasets. While it shows real-world examples, performance in extreme "white-out" blizzard conditions (where background information is almost entirely lost) is not explicitly detailed.
  • Generalization: The authors suggest the multi-scale projection idea is powerful; future work could involve applying this specific architecture to other adverse weather tasks like heavy rain or underwater image enhancement to see if the efficiency gains hold up.

Summary updated

Abstract

Snow removal causes challenges due to its characteristic of complex degradations. To this end, targeted treatment of multi-scale snow degradations is critical for the network to learn effective snow removal. In order to handle the diverse scenes, we propose a multi-scale projection transformer (MSP-Former), which understands and covers a variety of snow degradation features in a multi-path manner, and integrates comprehensive scene context information for clean reconstruction via self-attention operation. For the local details of various snow degradations, the local capture module is introduced in parallel to assist in the rebuilding of a clean image. Such design achieves the SOTA performance on three desnowing benchmark datasets while costing the low parameters and computational complexity, providing a guarantee of practicality.
BibTeX
@inproceedings{ye2023msp,
  author = {Tian Ye and Sixiang Chen and Tian Ye and Yun Liu and Taodong Liao and Jingxia Jiang and Erkang Chen and Peng Chen},
  title = {{MSP-Former: Multi-Scale Projection Transformer for Single Image Desnowing}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095605},
}
Figure