Controllable Weather Synthesis and Removal with Video Diffusion Models

Chih-Hao Lin, Zian Wang, Ruofan Liang, Yuxuan Zhang, Sanja Fidler, Shenlong Wang, Zan Gojcic

Abstract

WEATHERWEAVER is a novel generative framework designed to both synthesize and remove complex weather effects in videos with high precision. By decomposing the task into two specialized diffusion models and leveraging a multi-source data strategy, the system achieves photorealistic results for effects such as rain, snow, and fog. Notably, it successfully models persistent environmental changes like ground puddles and snow accumulation, surpassing existing video editing baselines in both temporal consistency and user preference.

Introduction

Simulating realistic weather in video sequences is a long-standing challenge with significant implications for film production, gaming, and the testing of autonomous driving systems. Traditionally, this has required complex physics-based simulations that are difficult to scale to diverse, real-world environments. While generative video editing models offer more flexibility, they often struggle with temporal flickering and lack the granular control necessary to adjust weather intensity.

Furthermore, the field faces a severe lack of paired training data; it is nearly impossible to capture the exact same real-world scene under varying weather conditions. WEATHERWEAVER addresses these challenges by treating weather as a controllable video-to-video translation task.

Methodology

WEATHERWEAVER utilizes a dual-model architecture built upon Stable Video Diffusion (SVD). The system simplifies the learning objective by passing all transformations through a "canonical" weather-free state.

Dual-Model Architecture

  1. Weather Removal Model: This component translates videos containing weather effects into a clear-sky version.
  2. Weather Synthesis Model: This model takes a clear video and adds weather effects based on a 6D control vector. This vector allows users to independently tune the intensity of six distinct attributes: clouds, fog, rain, snow, puddles, and snow coverage.

To incorporate these conditions, the researchers modified the SVD UNet by adding zero-initialized channels to the first layer, enabling the model to integrate the clear video and weather strength maps during training.

Overview of WEATHERWEAVER's capabilities, demonstrating both weather synthesis (left) and removal (right) for fog, snow, and rain.

Data Strategy

To overcome the scarcity of paired data, the authors employed a "triple threat" strategy:

  • Simulation: 20.8k video pairs from Unreal Engine provided perfect ground truth for weather intensity.
  • Generation: 1.1M image pairs were created using SDXL and Prompt-to-Prompt editing to ensure scene diversity.
  • Auto-labeled Real Videos: The team collected 4.6k real-world weather videos, used their own Removal Model to create "clear" versions, and employed a Vision Language Model (VLM) to estimate weather intensity, creating a self-improving training loop.

Experiments and Results

WEATHERWEAVER was evaluated against state-of-the-art baselines like TokenFlow and FRESCO using datasets such as Waymo Open Dataset and DL3DV.

  • Quantitative Performance: In user studies, WEATHERWEAVER was preferred over existing methods in 59% to 89% of synthesis cases and 52% to 89% of removal cases. It also achieved superior alignment scores with VLMs and CLIP, indicating that the generated effects accurately reflect the requested intensities.
  • Qualitative Quality: The model demonstrates exceptional temporal stability. Unlike previous methods that focus only on transient effects (e.g., falling rain), WEATHERWEAVER successfully models persistent effects. For instance, snow accumulation respects the underlying geometry of the scene, building up on flat surfaces while leaving vertical walls clear.

Conclusion

WEATHERWEAVER represents a significant step forward in controllable video editing. By decoupling synthesis and removal and utilizing a hybrid data approach, it provides a robust tool for creating and cleaning weather effects. While the model currently faces limitations regarding fine detail preservation due to VAE constraints and struggles with complex nighttime lighting, it sets a new benchmark for temporal consistency and physical realism in weather simulation.

Summary updated

BibTeX
@inproceedings{lin2025controllable,
  author = {Chih-Hao Lin and Zian Wang and Ruofan Liang and Yuxuan Zhang and Sanja Fidler and Shenlong Wang and Zan Gojcic},
  title = {{Controllable Weather Synthesis and Removal with Video Diffusion Models}},
  booktitle = {Wild3D @ ICCV},
  year = {2025},
  url = {https://openreview.net/forum?id=boos0tZP2U},
}
Figure