Introduction
Single-image deweathering—the task of removing rain, snow, or fog from a single photograph—is a critical component for autonomous systems and outdoor computer vision. However, the field is currently "dataset bottlenecked." Training effective deep learning models requires high-quality image pairs: one degraded by weather and a perfectly clean "ground truth" version of the exact same scene.
Historically, researchers have relied on two flawed alternatives. Synthetic datasets add "fake" weather effects to clean images, but often fail to capture complex real-world physics like volumetric scattering or the "veiling" effect of heavy rain. Conversely, manual collection involves humans photographing the same scene before and after weather events, a process that is labor-intensive, small-scale, and prone to lighting shifts. WeatherStream addresses this by automating the collection of real-world, time-multiplexed pairs at an unprecedented scale.
The WeatherStream Pipeline
The core innovation of WeatherStream is an automated pipeline that monitors over 5,000 outdoor YouTube live streams across 94 countries. To curate a reliable dataset from millions of raw frames, the system identifies "magic moments" where weather transitions occur rapidly without changes to the underlying scene geometry.
As shown in the pipeline diagram, the system employs several automated checks to ensure data quality. It uses optical flow to verify scene conformity (ensuring no cars or people moved between frames) and performs chromatic and illumination checks to reject frames where global lighting shifted significantly, such as moving clouds or sensor noise.
Physics-Based Filtering
Rather than relying solely on neural networks to identify weather, WeatherStream utilizes classical light-transport principles to verify the physical authenticity of the data. This ensures that the "weather" image is not just different, but specifically degraded according to the laws of physics.
The pipeline applies four specific Light Transport Principles:
- Background Conformity: Ensuring static scene geometry via optical flow.
- Particle Chromatic Variation: Verifying that weather artifacts show consistent color changes across RGB channels, consistent with the physics of falling rain or snow.
- Scatter-dependent Blur: Adverse weather acts as a low-pass filter. The pipeline uses Fast Fourier Transform (FFT) metrics to verify that the weather-degraded image is physically blurrier and lower-contrast than its clean counterpart.
- Illumination Consistency: A seed model ensures that the global radiance has not shifted, preventing false positives caused by changing time-of-day or cloud cover.
This rigorous filtering process rejects approximately 99.6% of candidate frames, resulting in a highly reliable dataset of 202,000 image pairs—the largest all-weather removal dataset currently available.
Key Results and Insights
The authors demonstrated that for deweathering, the quality and type of data are more important than the specific neural network architecture. By retraining several state-of-the-art models (such as Restormer and TransWeather) on the WeatherStream dataset, they achieved an average performance boost of 1.5 dB PSNR.
Qualitatively, models trained on WeatherStream are significantly better at capturing the "veiling effect"—the hazy, volumetric scattering common in real-world heavy rain that synthetic models often miss. Furthermore, because the automated pipeline ensures consistent lighting, the resulting models avoid the over-smoothing and color-shifting artifacts common in models trained on manually collected data.
Conclusion and Future Work
WeatherStream proves that automated, physics-informed data curation can overcome the limitations of both synthetic and manual datasets. While currently limited by the resolution of public webcams and the need for weather transitions, the project provides a scalable blueprint for real-world computer vision training. Future iterations aim to incorporate semantic segmentation to assist autonomous driving models in interpreting scenes through extreme weather conditions.
Summary updated