Selective Generative Restoration for Video Deraining via Uncertainty-Aware Diffusion

Yi'ang Chen, Han Li, Laibin Chang, Hui Liu, Kui Jiang, Junjun JiangarXiv 2026 · arXiv

TL;DR

Selective Generative Video Deraining (SGVD) is a novel framework that adaptively applies generative diffusion priors only to regions with high restoration uncertainty, such as heavy rain occlusions. By combining this uncertainty-aware fusion with a trajectory consistency mechanism, SGVD achieves state-of-the-art, flicker-free video restoration in a single inference step.

What Problem Does This Paper Address?

Removing rain from video is notoriously difficult because rain isn't uniform. Some regions might have light streaks where the background is still visible, while others are completely occluded by heavy "rain veils" or splashes.

Traditional deterministic models (like standard CNNs or Transformers) try to map every rainy pixel to a single "correct" clean pixel. In heavily occluded areas, this leads to blurry, over-smoothed results because the model doesn't have enough information to reconstruct the details. On the flip side, generative models (like standard Diffusion Models) are great at "hallucinating" missing details, but when applied globally, they often introduce fake textures in reliable regions where the original video was actually fine.

The authors identify this as a spatially heterogeneous inverse problem. The challenge is knowing where to trust the original pixels and where to let the generative model take over.

Fig. 1: Motivation of Selective Generative Restoration. Determin- istic restoration tends to produce over-smoothed details in severely degraded regions, while global generative restoration may introduce content hallucination in reliable regions. In contrast, the proposed paradigm performs region-adaptive restoration by preserving fidelity in low-uncertainty regions and selectively activating generative re- covery in high-uncertainty regions.

Methods

The proposed SGVD framework operates in the latent space of a pretrained Diffusion Transformer (DiT). Its core innovation lies in three components:

  1. Uncertainty-Guided Selective Generation (USG): To find the "hard" parts of a frame, SGVD runs two parallel one-step diffusion predictions with different noise seeds. If the two predictions disagree significantly in a certain region, that region is marked as "high uncertainty." The model then uses this uncertainty map to fuse the generative prediction with the original rainy features—generative priors dominate in high-uncertainty zones, while original fidelity is preserved in low-uncertainty zones.
  2. Anchor-Guided Trajectory Consistency (ATC): Generative models often suffer from "temporal flickering" because each frame might be hallucinated slightly differently. ATC regularizes the diffusion "trajectories" of neighboring frames by forcing them to maintain consistent relative distances to a shared five-frame temporal anchor.
  3. One-Step Efficiency: To make diffusion practical for video, the authors use consistency learning to compress the typical multi-step denoising process into a single efficient step.
Fig. 2: Overall architecture of the proposed SGVD for video deraining. The rainy video is first encoded into the latent space by a frozen VAE encoder. The Uncertainty-Guided Selective Generation (USG) module estimates a spatial uncertainty map from the discrepancy between dual-noise clean latent predictions, and adaptively combines the generative latent with the fidelity-preserving LQ latent. The resulting conditional latent is injected into the frozen DiT backbone through a lightweight trainable adapter. Meanwhile, Anchor- Guided Trajectory Consistency (ATC) regularizes each frame's distance to a shared five-frame anchor across trajectory states, promoting temporal coherence without evaluating this training loss at inference.

Datasets

The model was trained and evaluated on several standard benchmarks:

  • NTURain-Syn & NTURain-Real: A mix of synthetic rainy videos and real-world footage.
  • RainSynLight25 & RainSynComplex25: Datasets representing different levels of rain severity.
  • RVDT: Used specifically to evaluate how deraining improves downstream tasks like object detection and tracking.

Key Insights and Ideas

  • Stochastic Diversity as an Uncertainty Signal: One of the most elegant ideas here is using the "disagreement" between two random diffusion runs as a proxy for restoration difficulty. It’s a principled way to detect where the data is missing vs. where it is just noisy.
  • Selective > Global: The paper proves that "more generation" isn't always better. By restricting generative priors to high-uncertainty regions, they avoid the "uncanny valley" of hallucinated details in otherwise clear parts of the video.
  • Trajectory-Level Consistency: Instead of just penalizing pixel differences between frames (which can kill motion), they regularize the evolution of the latent states. This allows for natural motion while preventing the "drift" that causes flickering.

Key Results

SGVD outperforms both traditional and diffusion-based state-of-the-art (SOTA) methods across multiple metrics:

  • Fidelity & Perception: It achieves superior PSNR and SSIM while maintaining high perceptual scores (NIQE, CLIP-IQA). For example, on the NTURain-Syn dataset, it significantly improved perceptual quality over non-generative SOTA like RainMamba.
  • Temporal Stability: It achieved the lowest Warping Error (WE) among compared methods, indicating much smoother, more consistent video.
  • Efficiency: Thanks to the one-step formulation, SGVD is roughly 7x faster than standard 30-step diffusion models, making it far more viable for real-world applications.
  • Downstream Impact: Using SGVD as a pre-processor improved object detection (mAP) on rainy videos by over 2% compared to raw rainy input.

Limitations and Future Work

While SGVD is a major step forward, the authors note a few hurdles:

  • Computational Overhead: Even though it's a one-step model, running "dual probes" for uncertainty estimation effectively doubles the backbone computation during inference.
  • Complex Textures: The model still struggles with extremely fine textures and complex motion boundaries where uncertainty estimation might be less precise.
  • Future Directions: Reducing the cost of uncertainty estimation and further improving detail fidelity in high-motion scenes are the primary goals for future iterations.

Summary updated

Abstract

Video deraining is challenging due to rain streaks, occlusions, and atmospheric effects. Existing methods mainly follow a deterministic restoration paradigm, assuming that clean scenes can be uniquely recovered from rainy observations. However, rain degradation is spatially heterogeneous. Reliable regions require faithful reconstruction, whereas severely occluded regions contain ambiguous content that require stronger prior-driven inference. Therefore, conventional deterministic restoration often produces over-smoothed results, while global generative restoration may introduce unnecessary hallucinations. We introduce a Selective Generative Restoration paradigm for video deraining, where generative priors are adaptively activated according to restoration uncertainty. Based on this principle, we propose Selective Generative Video Deraining (SGVD), an uncertainty-aware diffusion framework that enables adaptive fidelity preservation and generative recovery. Specifically, SGVD exploits stochastic diversity between dual one-step diffusion predictions to estimate region-level restoration uncertainty, which determines where observation fidelity should be preserved and where generative inference should be activated. Furthermore, an anchor-guided trajectory consistency mechanism is proposed to regularize stochastic diffusion evolution across frames, reducing temporal generation drift and improving video coherence. To achieve practical efficiency, SGVD adopts one-step diffusion inference while maintaining uncertainty-aware generative capability. Extensive experiments demonstrate that SGVD achieves superior restoration performance, perceptual quality, and temporal consistency compared with state-of-the-art video deraining methods.
BibTeX
@misc{chen2026selective,
  author = {Yi'ang Chen and Han Li and Laibin Chang and Hui Liu and Kui Jiang and Junjun Jiang},
  title = {{Selective Generative Restoration for Video Deraining via Uncertainty-Aware Diffusion}},
  year = {2026},
  eprint = {2611.16310},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16310},
}
Figure