WeatherStream: Light Transport Automation of Single Image Deweathering

Howard Zhang, Yunhao Ba, Ethan Yang, Varan Mehra, Blake Gella, Akira Suzuki, Arnold Pfahnl, Chethan Chinder Chandrappa, Alex Wong, Achuta Kadambi

Introduction

Single-image deweathering—the task of removing rain, snow, or fog from a single photograph—is a critical component for autonomous systems and outdoor computer vision. However, the field is currently "dataset bottlenecked." Training effective deep learning models requires high-quality image pairs: one degraded by weather and a perfectly clean "ground truth" version of the exact same scene.

Historically, researchers have relied on two flawed alternatives. Synthetic datasets add "fake" weather effects to clean images, but often fail to capture complex real-world physics like volumetric scattering or the "veiling" effect of heavy rain. Conversely, manual collection involves humans photographing the same scene before and after weather events, a process that is labor-intensive, small-scale, and prone to lighting shifts. WeatherStream addresses this by automating the collection of real-world, time-multiplexed pairs at an unprecedented scale.

The WeatherStream Pipeline

The core innovation of WeatherStream is an automated pipeline that monitors over 5,000 outdoor YouTube live streams across 94 countries. To curate a reliable dataset from millions of raw frames, the system identifies "magic moments" where weather transitions occur rapidly without changes to the underlying scene geometry.

The WeatherStream automated curation pipeline, illustrating the filtering stages from raw video to final ground-truth pairs.

As shown in the pipeline diagram, the system employs several automated checks to ensure data quality. It uses optical flow to verify scene conformity (ensuring no cars or people moved between frames) and performs chromatic and illumination checks to reject frames where global lighting shifted significantly, such as moving clouds or sensor noise.

Physics-Based Filtering

Rather than relying solely on neural networks to identify weather, WeatherStream utilizes classical light-transport principles to verify the physical authenticity of the data. This ensures that the "weather" image is not just different, but specifically degraded according to the laws of physics.

Light transport model illustrating how particles cause scattering, airlight, and blur (APSF) in adverse weather.

The pipeline applies four specific Light Transport Principles:

  1. Background Conformity: Ensuring static scene geometry via optical flow.
  2. Particle Chromatic Variation: Verifying that weather artifacts show consistent color changes across RGB channels, consistent with the physics of falling rain or snow.
  3. Scatter-dependent Blur: Adverse weather acts as a low-pass filter. The pipeline uses Fast Fourier Transform (FFT) metrics to verify that the weather-degraded image is physically blurrier and lower-contrast than its clean counterpart.
  4. Illumination Consistency: A seed model ensures that the global radiance has not shifted, preventing false positives caused by changing time-of-day or cloud cover.

This rigorous filtering process rejects approximately 99.6% of candidate frames, resulting in a highly reliable dataset of 202,000 image pairs—the largest all-weather removal dataset currently available.

Key Results and Insights

The authors demonstrated that for deweathering, the quality and type of data are more important than the specific neural network architecture. By retraining several state-of-the-art models (such as Restormer and TransWeather) on the WeatherStream dataset, they achieved an average performance boost of 1.5 dB PSNR.

Qualitatively, models trained on WeatherStream are significantly better at capturing the "veiling effect"—the hazy, volumetric scattering common in real-world heavy rain that synthetic models often miss. Furthermore, because the automated pipeline ensures consistent lighting, the resulting models avoid the over-smoothing and color-shifting artifacts common in models trained on manually collected data.

Conclusion and Future Work

WeatherStream proves that automated, physics-informed data curation can overcome the limitations of both synthetic and manual datasets. While currently limited by the resolution of public webcams and the need for weather transitions, the project provides a scalable blueprint for real-world computer vision training. Future iterations aim to incorporate semantic segmentation to assist autonomous driving models in interpreting scenes through extreme weather conditions.

Summary updated

Abstract

Today single image deweathering is arguably more sensitive to the dataset type, rather than the model. We introduce WeatherStream, an automatic pipeline capturing all real-world weather effects (rain, snow, and rain fog degradations), along with their clean image pairs. Previous state-of-the-art methods that have attempted the all-weather removal task train on synthetic pairs, and are thus limited by the Sim2Real domain gap. Recent work has attempted to manually collect time multiplexed pairs, but the use of human labor limits the scale of such a dataset. We introduce a pipeline that uses the power of light-transport physics and a model trained on a small, initial seed dataset to reject approximately 99.6% of unwanted scenes. The pipeline is able to generalize to new scenes and degradations that can, in turn, be used to train existing models just like fully human-labeled data. Training on a dataset collected through this procedure leads to significant improvements on multiple existing weather removal methods on a carefully human-collected test set of real-world weather effects. The dataset and code can be found in the following website: http://visual.ee.ucla.edu/wstream.htm/.
BibTeX
@inproceedings{zhang2023weatherstream,
  author = {Howard Zhang and Yunhao Ba and Ethan Yang and Varan Mehra and Blake Gella and Akira Suzuki and Arnold Pfahnl and Chethan Chinder Chandrappa and Alex Wong and Achuta Kadambi},
  title = {{WeatherStream: Light Transport Automation of Single Image Deweathering}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01297},
}

Loading related content...

Figure