TL;DR
LMQFormer is a lightweight, high-efficiency transformer model designed to remove snow from images in real-time. By using a novel Laplace-guided mask prior and a specialized "Duplicated Mask Query Attention" mechanism, it achieves state-of-the-art restoration quality with only 2.18 million parameters—making it up to 100x smaller and significantly faster than previous leading models.
What Problem Does This Paper Address?
Snow removal is significantly more complex than rain removal. While rain is often semi-transparent and follows regular patterns, snow is opaque, varies wildly in shape (from tiny flakes to large clumps), and often creates a "veiling effect" (haze) that obscures the background entirely.
Existing state-of-the-art (SOTA) models are computationally heavy, often requiring hundreds of millions of parameters to achieve good results. This makes them impractical for real-world deployment on edge devices, such as security cameras or autonomous vehicle sensors. Furthermore, many models leave "repairing traces"—visible artifacts where the snow used to be. The authors aim to bridge the gap between high-quality restoration and lightweight, real-time performance.
Methods
The authors introduce LMQFormer, which consists of two primary sub-networks:
- Laplace-VQVAE: This module generates a "coarse mask" of the snowy areas. Instead of relying on ground-truth masks (which aren't available for real-world images), it uses a Laplace operator to highlight high-frequency edges—effectively "finding" the snow. A Vector Quantized Variational AutoEncoder (VQVAE) then processes this to create a low-entropy prior that guides the rest of the network.
- MQFormer (Mask Query Transformer): This is the main recovery engine. It uses a dual-encoder architecture: one branch uses transformers to capture global context, while the other uses convolutions for local details.
- DMQA (Duplicated Mask Query Attention): This is the core innovation. Standard transformers calculate attention across the entire image, which is computationally expensive. DMQA uses the coarse mask to generate a specific number of "queries" that constrain the transformer's attention specifically to snowy regions. This drastically reduces the parameter count while maintaining high precision.
Datasets
The model was evaluated on a comprehensive suite of synthetic and real-world benchmarks:
- Synthetic: Snow100K (the standard large-scale dataset), SRRS (focuses on the veiling effect), CSD (focuses on snow streaks), and street-view datasets like SnowKITTI2012 and SnowCityScapes.
- Real-World: The Snow100K real-world test set, consisting of 1,329 images of actual snowy scenes to test generalization.
Key Insights and Ideas
- Entropy as a Bottleneck: The authors argue that snow adds "information entropy" to an image. By using the Laplace operator to isolate snow edges early, they reduce the complexity the recovery network has to deal with, allowing for a much smaller model.
- Masks as Constraints, Not Just Inputs: In many models, a mask is just another input layer. Here, the mask actively shapes the Attention mechanism. By forcing the "queries" in the transformer to align with the mask, the network doesn't waste "brainpower" (parameters) on parts of the image that are already clean.
- Hybrid Architecture: Combining the strengths of Transformers (global relationship modeling) and Convolutions (local feature extraction) at low scales allows the model to be efficient without losing the fine details needed to reconstruct the background.
Key Results
- Efficiency: LMQFormer uses only 2.18M parameters. For comparison, the previous SOTA model, DDMSNet, uses 229.45M—making LMQFormer roughly 100 times smaller.
- Speed: It achieves a runtime of 0.042 seconds per image, the fastest among all compared SOTA methods (TKL, the next fastest, takes 0.051s but has 13x more parameters).
- Quality: Despite its size, it outperformed larger models in PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity). On the Snow100K dataset, it achieved a PSNR of 31.88 dB, surpassing the next best model (TKL) at 30.51 dB.
- Visual Fidelity: Qualitative results show that LMQFormer is particularly good at avoiding artifacts on bright red objects (like traffic signs), a common failure point for other desnowing algorithms.
Limitations and Future Work
While the paper demonstrates excellent results, a few areas remain for exploration:
- Extreme Occlusion: The paper focuses on "lightweight" removal; it is unclear how the model performs in "white-out" conditions where the background is 100% occluded and must be hallucinated rather than just cleaned.
- Temporal Consistency: The current model processes single frames. Extending this lightweight architecture to video (ensuring snow removal doesn't "flicker" between frames) is a logical next step for applications in autonomous driving.
- Hardware-Specific Optimization: While the parameter count is low, the authors suggest further work could involve optimizing the specific transformer operations for mobile NPU (Neural Processing Unit) architectures.
Summary updated