Abstract: Videos captured in the wild often suffer from rain streaks, blur, and noise. In addition, even slight changes in camera pose can amplify cross-frame mismatches and temporal artifacts. Existing methods rely on optical flow or heuristic alignment, which are computationally expensive and less robust. To address these challenges, Lie groups provide a principled way to represent continuous geometric transformations, making them well-suited for enforcing spatial and temporal consistency in video modeling. Building on this insight, we propose DeLiVR, an efficient video deraining method that injects spatiotemporal Lie-group differential biases directly into attention scores of the network. Specifically, the method introduces two complementary components. First, a rotation-bounded Lie relative bias predicts the in-plane angle of each frame using a compact prediction module, where normalized coordinates are rotated and compared with base coordinates to achieve geometry-consistent alignment before feature aggregation. Second, a differential group displacement computes angular differences between adjacent frames to estimate a velocity. This bias computation combines temporal decay and attention masks to focus on inter-frame relationships while precisely matching the direction of rain streaks. Extensive experimental results demonstrate the effectiveness of our method on publicly available benchmarks.
Introduction
Removing rain from video is significantly more challenging than single-image deraining. Effective restoration requires aggregating information across multiple frames to distinguish transient rain streaks from static or moving scene details. Traditionally, this process relies on pixel-level alignment via optical flow. However, optical flow assumes "brightness constancy," a principle frequently violated by flickering rain streaks and occlusions.
When flow-based alignment fails, the resulting video often exhibits "ghosting" artifacts or temporal flickering. Furthermore, these methods are computationally intensive and struggle with the subtle camera shakes typical of handheld mobile footage. DeLiVR addresses these issues by moving away from dense flow fields toward a group-theoretic representation of motion.
Methodology
DeLiVR (Differential Spatiotemporal Lie Bias) introduces a mathematically principled approach to video deraining by injecting geometric biases directly into a Transformer’s attention mechanism. The architecture centers on three core innovations:
1. The SO(2) Head and Lie Algebra
Instead of predicting a dense flow field, DeLiVR employs a lightweight module to predict a single rotation matrix ($R_t$) for each frame using the Special Orthogonal group $SO(2)$. This captures global in-plane rotation and camera shake efficiently.
2. Spatiotemporal Lie Biases
- Rotation-Bounded Lie Relative Bias (Spatial): The model embeds spatial locations into a coordinate basis and rotates them using the predicted $SO(2)$ matrices. By computing a spatial bias based on the similarity of these rotated coordinates, the Transformer’s attention mechanism is guided to look at geometrically aligned patches across frames.
- Differential Group Displacement (Temporal): By projecting the difference between consecutive rotations onto the Lie algebra (the tangent space of the rotation group), the model estimates a "Lie-velocity." This velocity informs a temporal bias that penalizes erratic motion jumps, enforcing temporal smoothness.
3. Unified Attention Injection
These spatial and temporal biases are fused with a temporal decay mask and added directly to the attention scores of a standard Transformer backbone. This guides the network to focus on reliable correspondences while ignoring the chaotic noise of rain.
Experimental Results
The model was evaluated on four major benchmarks, including synthetic datasets (NTURain, Rain-Syn-Light/Complex) and the real-world WeatherBench. DeLiVR demonstrated superior performance and efficiency across several metrics:
- State-of-the-Art Performance: On the real-world WeatherBench dataset, DeLiVR achieved a PSNR of 26.56 dB, significantly outperforming the previous state-of-the-art, VDMamba (23.91 dB).
- Efficiency: DeLiVR is remarkably lightweight, utilizing only 2.64 million parameters—over 20 times fewer than competing high-performance models like Turtle (58.6M) or ViWS-Net (57.8M).
- Inference Speed: The model processes frames at approximately 82.5ms per frame, nearly twice as fast as flow-based competitors like rainmanba (145ms).
- Generalization: Beyond deraining, the method proved effective for deblurring (GoPro dataset) and dehazing (RESIDE dataset), suggesting the geometric bias is a robust, degradation-agnostic prior.
Conclusion
DeLiVR demonstrates that global geometric priors derived from Lie groups are more robust than local, error-prone optical flow for video restoration in heavily corrupted scenes. While currently focused on in-plane rotations ($SO(2)$), future work may extend this framework to richer transformation groups like $SE(3)$ to handle complex 3D camera translations and non-rigid deformations.
Summary updated