LMQFormer: A Laplace-Prior-Guided Mask Query Transformer for Lightweight Snow Removal

Junhong Lin, Nanfeng Jiang, Zhentao Zhang, Weiling Chen, Tiesong Zhao

Abstract

Snow removal is a critical task in computer vision, aimed at locating snow-covered areas and recovering clean background images. Unlike rain, snow presents diverse patterns and opaque degradations that heavily occlude the background, often requiring computationally expensive models for effective restoration. In this paper, we propose LMQFormer, a lightweight yet high-efficiency snow removal network. Our approach integrates a Laplace-VQVAE to generate a coarse mask as prior knowledge, reducing information entropy and computational costs. We further design a Mask Query Transformer (MQFormer) with a novel Duplicated Mask Query Attention (DMQA) mechanism that constrains attention to snowy regions, drastically reducing parameter counts. Experimental results demonstrate that LMQFormer achieves state-of-the-art quality on popular datasets with significantly fewer parameters and the lowest running time among compared methods.

The Challenge of Snow Removal

Snow removal is significantly more complex than rain removal. While rain is often semi-transparent and follows regular patterns, snow is opaque, varies wildly in shape—from tiny flakes to large clumps—and often creates a "veiling effect" (haze) that obscures the background entirely.

Existing state-of-the-art (SOTA) models are typically computationally heavy, often requiring hundreds of millions of parameters to achieve acceptable results. This makes them impractical for real-world deployment on edge devices, such as security cameras or autonomous vehicle sensors. Furthermore, many models leave "repairing traces"—visible artifacts where the snow used to be. LMQFormer aims to bridge the gap between high-quality restoration and lightweight, real-time performance.

Methodology: LMQFormer Architecture

LMQFormer consists of three primary innovations designed for efficiency:

  1. Laplace-VQVAE: This module generates a "coarse mask" of snowy areas. Instead of relying on ground-truth masks (unavailable for real-world images), it uses a Laplace operator to highlight high-frequency edges. A Vector Quantized Variational AutoEncoder (VQVAE) then processes this to create a low-entropy prior that guides the recovery network.
  2. MQFormer (Mask Query Transformer): The main recovery engine uses a dual-encoder architecture. One branch employs transformers to capture global context, while the other uses convolutions for local detail extraction.
  3. DMQA (Duplicated Mask Query Attention): Standard transformers calculate attention across the entire image, which is computationally expensive. DMQA uses the coarse mask to generate a specific number of "queries" that constrain the transformer's attention specifically to snowy regions. This drastically reduces the parameter count while maintaining high precision.

Key Insights

  • Entropy Reduction: By using the Laplace operator to isolate snow edges early, the network reduces the complexity of the recovery task, allowing for a much smaller model footprint.
  • Masks as Constraints: In LMQFormer, the mask is not just an input layer but actively shapes the attention mechanism. By forcing queries to align with the mask, the network avoids wasting parameters on clean parts of the image.
  • Hybrid Architecture: Combining Transformers for global relationships and Convolutions for local features at low scales allows the model to be efficient without losing the fine details needed for background reconstruction.

Experimental Performance

LMQFormer sets a new benchmark for efficiency in snow removal, as shown in the comparison below.

Qualitative and quantitative comparison on the Snow100K dataset. Plot (e) illustrates the superior trade-off between SSIM and parameter count achieved by LMQFormer (Ours).
  • Efficiency: LMQFormer uses only 2.18M parameters. For comparison, the previous SOTA model, DDMSNet, uses 229.45M—making LMQFormer roughly 100 times smaller.
  • Speed: It achieves a runtime of 0.042 seconds per image, the fastest among all compared SOTA methods (TKL, the next fastest, takes 0.051s but has 13x more parameters).
  • Quality: Despite its size, it outperformed larger models in PSNR and SSIM. On the Snow100K dataset, it achieved a PSNR of 31.88 dB, surpassing the next best model (TKL) at 30.51 dB.
  • Visual Fidelity: Qualitative results show that LMQFormer is particularly effective at avoiding artifacts on bright red objects (like traffic signs), a common failure point for other desnowing algorithms.

Conclusion

LMQFormer demonstrates that high-quality snow removal does not require massive parameter counts. By leveraging Laplace-guided priors and mask-constrained attention, it achieves real-time performance suitable for edge deployment. Future work will explore extending this architecture to video for temporal consistency in applications like autonomous driving.

Summary updated

Abstract

Snow removal aims to locate snow areas and recover clean images without repairing traces. Unlike the regularity and semitransparency of rain, snow with various patterns and degradations seriously occludes the background. As a result, the state-of-the-art snow removal methods usually retains a large parameter size. In this paper, we propose a lightweight but high-efficient snow removal network called Laplace Mask Query Transformer (LMQFormer). Firstly, we present a Laplace-VQVAE to generate a coarse mask as prior knowledge of snow. Instead of using the mask in dataset, we aim at reducing both the information entropy of snow and the computational cost of recovery. Secondly, we design a Mask Query Transformer (MQFormer) to remove snow with the coarse mask, where we use two parallel encoders and a hybrid decoder to learn extensive snow features under lightweight requirements. Thirdly, we develop a Duplicated Mask Query Attention (DMQA) that converts the coarse mask into a specific number of queries, which constraint the attention areas of MQFormer with reduced parameters. Experimental results in popular datasets have demonstrated the efficiency of our proposed model, which achieves the state-of-the-art snow removal quality with significantly reduced parameters and the lowest running time.
BibTeX
@article{lin2023lmqformer,
  author = {Junhong Lin and Nanfeng Jiang and Zhentao Zhang and Weiling Chen and Tiesong Zhao},
  title = {{LMQFormer: A Laplace-Prior-Guided Mask Query Transformer for Lightweight Snow Removal}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2023},
  url = {https://openreview.net/forum?id=AM3KFOqavE},
}
Figure