TL;DR
FluidRain introduces a physics-inspired video deraining framework that models rain streaks as an incompressible fluid flow. By using this divergence-free flow to guide an anisotropic attention mechanism within a highly parameter-efficient "Loop-in-Loop" architecture, it achieves state-of-the-art restoration performance with a tiny footprint of just 0.80M parameters.
What Problem Does This Paper Address?
Removing rain from video sequences (video deraining) is a classic computer vision challenge. Existing methods typically fall into two camps:
- Explicit alignment: These use optical flow to align neighboring frames before removing rain. However, optical flow estimation often breaks down under heavy, dense rain.
- Implicit aggregation: These avoid explicit alignment by letting deep networks fuse spatiotemporal features implicitly. Unfortunately, they lack explicit physical guidance regarding the directional and temporally coherent structure of rain.
This leaves a critical gap: how do we achieve reliable temporal aggregation without relying on brittle motion estimation? FluidRain tackles this by using principles of fluid mechanics to explicitly model rain motion without requiring explicit frame-to-frame pixel alignment.
Methods
FluidRain introduces two core innovations: a fluid-mechanical rain-flow representation and a weight-tied "Loop-in-Loop" architecture.
1. Divergence-Free Rain Flow & Attention Bias
Instead of tracking scene correspondences (like optical flow), the authors model the coherent motion of rain within a frame as an image-space velocity field, termed rain flow. Inspired by fluid mechanics, they enforce an incompressibility constraint by projecting raw neural flow predictions onto a divergence-free subspace ($\nabla \cdot \mathbf{u} = 0$).
This clean rain flow determines the local orientation and length of rain streaks. Rather than warping features, FluidRain converts these physical properties into an anisotropic attention bias. This additive bias mathematically steers the network's window attention along the rain streaks, allowing it to aggregate clean contextual information from neighboring regions and frames without explicit alignment.
2. Loop-in-Loop Architecture
To keep the model lightweight, the authors exploit two physical regularities of rain:
- Short-term persistence (Time): Rain flow structures vary slowly across adjacent frames, even though individual streaks change.
- Depth-scale equivariance (Scale): Downsampling an image rescales the length of a rain streak but preserves its direction.
These properties motivate the Loop-in-Loop design:
- Inner Loop: Reuses the exact same attention operator to read keys and values across neighboring frames ($\{t-1, t, t+1\}$).
- Outer Loop: Reuses the same block stack across three spatial resolutions (scales $1$, $1/2$, and $1/4$).
By repeatedly recycling the same weight-tied parameters across both time and scale, the entire network requires only 0.80M parameters.
Datasets
The model was trained and evaluated on four standard synthetic video deraining benchmarks:
- NTURain
- RainSynLight
- RainSynComplex
- RainSynAll100
New Benchmarks & Metrics Introduced:
- RainSyn-Gust: Existing benchmarks assume a constant rain direction throughout a video. To test real-world robustness, the authors introduced RainSyn-Gust, which injects controlled changes in rain-streak orientation—simulating sudden camera movements or wind gusts.
- $F_{phys}$ Metric: Real-world rainy videos lack clean ground-truth references. The authors developed a physics-based, no-reference metric ($F_{phys}$) that balances precision (whether the removed content physically behaves like rain) and recall (how much rain-like energy was removed).
Key Insights and Ideas
- Rain as an Incompressible Fluid: Projecting the estimated rain flow onto a divergence-free subspace acts as a powerful physical regularizer. It filters out unphysical neural network predictions, ensuring the attention mechanism is guided by true atmospheric physics.
- Steering vs. Warping: Instead of trying to perfectly align rain-corrupted pixels across frames (which is error-prone), it is far more effective to steer the attention mechanism along the physical path of the rain streaks to find uncorrupted background details.
- Recurrence over Depth: Physical symmetries in data (like scale-invariant direction) mean that deep networks do not always need unique parameters for every layer. Massive parameter sharing via looped architectures can match the capacity of much larger models.
Key Results
Despite having only 0.80M parameters—compared to heavyweights like Restormer (26.11M) and ESTINet (29.90M)—FluidRain delivers highly competitive performance:
- Quantitative Performance: It achieved the second-highest PSNR on both NTURain (35.73 dB) and RainSynComplex (30.18 dB), outperforming several models more than 30 times its size.
- Robustness to Dynamics: On the challenging RainSyn-Gust dataset, standard per-frame models suffered severe performance drops when rain direction shifted. FluidRain remained stable due to its direction-aware attention guidance.
- Downstream Tasks: When tested on KITTI tracking and segmentation datasets under rainy conditions, preprocessing videos with FluidRain recovered 25% of the rain-induced degradation for object detection and 52% for semantic segmentation.
Limitations and Future Work
While highly efficient, FluidRain operates on a localized three-frame temporal window ($\{t-1, t, t+1\}$). This design choice means it cannot capture long-range temporal dependencies or historical context spanning dozens of frames, which newer State-Space Models (SSMs) or long-form recurrent networks can exploit. Extending the Loop-in-Loop recurrence to handle longer video clips without a linear increase in computational overhead remains an open avenue for future research.
Summary updated