Single Image Snow Removal via Composition Generative Adversarial Networks

Zhi Li, Juan Zhang, Zhijun Fang, Bo Huang, Xiaoyan Jiang, Yongbin Gao, Jenq-Neng HwangIEEE Access 2019 · DOI

Introduction

Removing snow from a single image is a challenging inverse problem in computer vision. Unlike rain streaks, which are typically thin and translucent, snowflakes are often large, opaque, and vary significantly in shape and density. When a snowflake occludes a scene, the background information at those pixels is effectively lost. Existing restoration methods frequently suffer from visual artifacts or unnatural color shifts, failing to fully recover the underlying scene in heavy snow conditions.

To address these challenges, researchers have developed the Composition Generative Adversarial Network (CGAN). This architecture leverages a physics-informed approach to simultaneously reconstruct the clean background and estimate the snow mask, ensuring the final output remains faithful to the original scene's geometry and color.

The CGAN Architecture

The core of the CGAN is a specialized generator composed of two distinct modules:

  1. Clean Background Module: An encoder-decoder network designed to predict the latent "clean" image.
  2. Snow Mask Estimate Module: A sub-network that identifies the precise location and intensity of snowflakes within the input image.

By decoupling these tasks, the model can intelligently decide where to fill in missing data (inpainting) versus where to preserve existing background details.

Physics-Informed Composition Loss

The most significant innovation is the introduction of a Composition Loss. This loss function is based on a mathematical model of snowy image formation:

$$I = J \odot m + A \odot (1 - m)$$

where $I$ is the observed snowy image, $J$ represents the snow, $m$ is the snow mask, and $A$ is the clean background. During training, the network is constrained to ensure that the predicted background and mask can be mathematically recombined to recreate the original snowy input. This self-supervisory constraint prevents the model from "hallucinating" unrealistic details and helps maintain global color consistency.

Training and Implementation

The model utilizes a Least Squares GAN (LSGAN) loss function to improve training stability and image quality compared to standard GAN objectives. Furthermore, a 70x70 PatchGAN discriminator is employed to focus on local image patches, which helps the model preserve fine textures and sharp edges in the reconstructed background. The researchers also found that Mean Absolute Error (MAE) loss was superior to standard MSE for encouraging the background module to stay close to ground truth without blurring edges.

The system was trained on the Snow100K dataset, which contains 100,000 synthetic snowy images paired with ground-truth clean images and snow masks. Testing on real-world snowy photos confirmed that the synthetic training generalizes well to actual weather conditions.

Key Results and Performance

The CGAN achieves state-of-the-art performance, outperforming previous benchmarks like DesnowNet. On the Snow100K test set, the model reached a PSNR of 30.39 and an SSIM of 0.935.

Qualitative comparison of snowy input images (left) and the CGAN's desnowed results (right) across various scenes.

Beyond numerical metrics, the qualitative improvements are stark. The CGAN is notably better at preserving distant light sources and fine architectural details that are often blurred by other methods. Additionally, the researchers demonstrated that pre-processing snowy images with CGAN significantly improves the accuracy of downstream tasks, such as object detection using YOLO-V3.

Conclusion and Future Work

By incorporating the physics of image formation into a deep learning framework, CGAN provides a robust solution for single-image snow removal. While the model occasionally struggles with extremely heavy "wet" snow that smears across lenses, it represents a major step forward in weather-robust computer vision. Future research may focus on unsupervised domain adaptation to further bridge the gap between synthetic training data and the diverse characteristics of real-world snowfall.

Summary updated

BibTeX
@article{li2019single,
  author = {Zhi Li and Juan Zhang and Zhijun Fang and Bo Huang and Xiaoyan Jiang and Yongbin Gao and Jenq-Neng Hwang},
  title = {{Single Image Snow Removal via Composition Generative Adversarial Networks}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2900323},
}
Figure