TL;DR
Selective Generative Video Deraining (SGVD) is a novel framework that adaptively applies generative diffusion priors only to regions with high restoration uncertainty, such as heavy rain occlusions. By combining this uncertainty-aware fusion with a trajectory consistency mechanism, SGVD achieves state-of-the-art, flicker-free video restoration in a single inference step.
What Problem Does This Paper Address?
Removing rain from video is notoriously difficult because rain isn't uniform. Some regions might have light streaks where the background is still visible, while others are completely occluded by heavy "rain veils" or splashes.
Traditional deterministic models (like standard CNNs or Transformers) try to map every rainy pixel to a single "correct" clean pixel. In heavily occluded areas, this leads to blurry, over-smoothed results because the model doesn't have enough information to reconstruct the details. On the flip side, generative models (like standard Diffusion Models) are great at "hallucinating" missing details, but when applied globally, they often introduce fake textures in reliable regions where the original video was actually fine.
The authors identify this as a spatially heterogeneous inverse problem. The challenge is knowing where to trust the original pixels and where to let the generative model take over.
Methods
The proposed SGVD framework operates in the latent space of a pretrained Diffusion Transformer (DiT). Its core innovation lies in three components:
- Uncertainty-Guided Selective Generation (USG): To find the "hard" parts of a frame, SGVD runs two parallel one-step diffusion predictions with different noise seeds. If the two predictions disagree significantly in a certain region, that region is marked as "high uncertainty." The model then uses this uncertainty map to fuse the generative prediction with the original rainy features—generative priors dominate in high-uncertainty zones, while original fidelity is preserved in low-uncertainty zones.
- Anchor-Guided Trajectory Consistency (ATC): Generative models often suffer from "temporal flickering" because each frame might be hallucinated slightly differently. ATC regularizes the diffusion "trajectories" of neighboring frames by forcing them to maintain consistent relative distances to a shared five-frame temporal anchor.
- One-Step Efficiency: To make diffusion practical for video, the authors use consistency learning to compress the typical multi-step denoising process into a single efficient step.
Datasets
The model was trained and evaluated on several standard benchmarks:
- NTURain-Syn & NTURain-Real: A mix of synthetic rainy videos and real-world footage.
- RainSynLight25 & RainSynComplex25: Datasets representing different levels of rain severity.
- RVDT: Used specifically to evaluate how deraining improves downstream tasks like object detection and tracking.
Key Insights and Ideas
- Stochastic Diversity as an Uncertainty Signal: One of the most elegant ideas here is using the "disagreement" between two random diffusion runs as a proxy for restoration difficulty. It’s a principled way to detect where the data is missing vs. where it is just noisy.
- Selective > Global: The paper proves that "more generation" isn't always better. By restricting generative priors to high-uncertainty regions, they avoid the "uncanny valley" of hallucinated details in otherwise clear parts of the video.
- Trajectory-Level Consistency: Instead of just penalizing pixel differences between frames (which can kill motion), they regularize the evolution of the latent states. This allows for natural motion while preventing the "drift" that causes flickering.
Key Results
SGVD outperforms both traditional and diffusion-based state-of-the-art (SOTA) methods across multiple metrics:
- Fidelity & Perception: It achieves superior PSNR and SSIM while maintaining high perceptual scores (NIQE, CLIP-IQA). For example, on the NTURain-Syn dataset, it significantly improved perceptual quality over non-generative SOTA like RainMamba.
- Temporal Stability: It achieved the lowest Warping Error (WE) among compared methods, indicating much smoother, more consistent video.
- Efficiency: Thanks to the one-step formulation, SGVD is roughly 7x faster than standard 30-step diffusion models, making it far more viable for real-world applications.
- Downstream Impact: Using SGVD as a pre-processor improved object detection (mAP) on rainy videos by over 2% compared to raw rainy input.
Limitations and Future Work
While SGVD is a major step forward, the authors note a few hurdles:
- Computational Overhead: Even though it's a one-step model, running "dual probes" for uncertainty estimation effectively doubles the backbone computation during inference.
- Complex Textures: The model still struggles with extremely fine textures and complex motion boundaries where uncertainty estimation might be less precise.
- Future Directions: Reducing the cost of uncertainty estimation and further improving detail fidelity in high-motion scenes are the primary goals for future iterations.
Summary updated