DeLiVR: Differential Spatiotemporal Lie Bias for Efficient Video Deraining

Shuning Sun, Jialang Lu, Xiang Chen, Jichao Wang, Dianjie Lu, Guijuan Zhang, Guangwei Gao, Zhuoran Zheng

Abstract: Videos captured in the wild often suffer from rain streaks, blur, and noise. In addition, even slight changes in camera pose can amplify cross-frame mismatches and temporal artifacts. Existing methods rely on optical flow or heuristic alignment, which are computationally expensive and less robust. To address these challenges, Lie groups provide a principled way to represent continuous geometric transformations, making them well-suited for enforcing spatial and temporal consistency in video modeling. Building on this insight, we propose DeLiVR, an efficient video deraining method that injects spatiotemporal Lie-group differential biases directly into attention scores of the network. Specifically, the method introduces two complementary components. First, a rotation-bounded Lie relative bias predicts the in-plane angle of each frame using a compact prediction module, where normalized coordinates are rotated and compared with base coordinates to achieve geometry-consistent alignment before feature aggregation. Second, a differential group displacement computes angular differences between adjacent frames to estimate a velocity. This bias computation combines temporal decay and attention masks to focus on inter-frame relationships while precisely matching the direction of rain streaks. Extensive experimental results demonstrate the effectiveness of our method on publicly available benchmarks.

Introduction

Removing rain from video is significantly more challenging than single-image deraining. Effective restoration requires aggregating information across multiple frames to distinguish transient rain streaks from static or moving scene details. Traditionally, this process relies on pixel-level alignment via optical flow. However, optical flow assumes "brightness constancy," a principle frequently violated by flickering rain streaks and occlusions.

When flow-based alignment fails, the resulting video often exhibits "ghosting" artifacts or temporal flickering. Furthermore, these methods are computationally intensive and struggle with the subtle camera shakes typical of handheld mobile footage. DeLiVR addresses these issues by moving away from dense flow fields toward a group-theoretic representation of motion.

Methodology

DeLiVR (Differential Spatiotemporal Lie Bias) introduces a mathematically principled approach to video deraining by injecting geometric biases directly into a Transformer’s attention mechanism. The architecture centers on three core innovations:

1. The SO(2) Head and Lie Algebra

Instead of predicting a dense flow field, DeLiVR employs a lightweight module to predict a single rotation matrix ($R_t$) for each frame using the Special Orthogonal group $SO(2)$. This captures global in-plane rotation and camera shake efficiently.

The SO(2) prediction head maps frame features to the Lie algebra to generate rotation matrices via the exponential map.

2. Spatiotemporal Lie Biases

  • Rotation-Bounded Lie Relative Bias (Spatial): The model embeds spatial locations into a coordinate basis and rotates them using the predicted $SO(2)$ matrices. By computing a spatial bias based on the similarity of these rotated coordinates, the Transformer’s attention mechanism is guided to look at geometrically aligned patches across frames.
  • Differential Group Displacement (Temporal): By projecting the difference between consecutive rotations onto the Lie algebra (the tangent space of the rotation group), the model estimates a "Lie-velocity." This velocity informs a temporal bias that penalizes erratic motion jumps, enforcing temporal smoothness.
The Lie-equivalent predictor aligns tilted image patches into a canonical orientation for consistent feature aggregation.

3. Unified Attention Injection

These spatial and temporal biases are fused with a temporal decay mask and added directly to the attention scores of a standard Transformer backbone. This guides the network to focus on reliable correspondences while ignoring the chaotic noise of rain.

Experimental Results

The model was evaluated on four major benchmarks, including synthetic datasets (NTURain, Rain-Syn-Light/Complex) and the real-world WeatherBench. DeLiVR demonstrated superior performance and efficiency across several metrics:

  • State-of-the-Art Performance: On the real-world WeatherBench dataset, DeLiVR achieved a PSNR of 26.56 dB, significantly outperforming the previous state-of-the-art, VDMamba (23.91 dB).
  • Efficiency: DeLiVR is remarkably lightweight, utilizing only 2.64 million parameters—over 20 times fewer than competing high-performance models like Turtle (58.6M) or ViWS-Net (57.8M).
  • Inference Speed: The model processes frames at approximately 82.5ms per frame, nearly twice as fast as flow-based competitors like rainmanba (145ms).
  • Generalization: Beyond deraining, the method proved effective for deblurring (GoPro dataset) and dehazing (RESIDE dataset), suggesting the geometric bias is a robust, degradation-agnostic prior.

Conclusion

DeLiVR demonstrates that global geometric priors derived from Lie groups are more robust than local, error-prone optical flow for video restoration in heavily corrupted scenes. While currently focused on in-plane rotations ($SO(2)$), future work may extend this framework to richer transformation groups like $SE(3)$ to handle complex 3D camera translations and non-rigid deformations.

Summary updated

Abstract

Videos captured in the wild often suffer from rain streaks, blur, and noise. In addition, even slight changes in camera pose can amplify cross-frame mismatches and temporal artifacts. Existing methods rely on optical flow or heuristic alignment, which are computationally expensive and less robust. To address these challenges, Lie groups provide a principled way to represent continuous geometric transformations, making them well-suited for enforcing spatial and temporal consistency in video modeling. Building on this insight, we propose DeLiVR, an efficient video deraining method that injects spatiotemporal Lie-group differential biases directly into attention scores of the network. Specifically, the method introduces two complementary components. First, a rotation-bounded Lie relative bias predicts the in-plane angle of each frame using a compact prediction module, where normalized coordinates are rotated and compared with base coordinates to achieve geometry-consistent alignment before feature aggregation. Second, a differential group displacement computes angular differences between adjacent frames to estimate a velocity. This bias computation combines temporal decay and attention masks to focus on inter-frame relationships while precisely matching the direction of rain streaks. Extensive experimental results demonstrate the effectiveness of our method on publicly available benchmarks. The code is publicly available at https://github.com/Shuning0312/ICLR-DeLiVR.
BibTeX
@inproceedings{sun2026delivr,
  author = {Shuning Sun and Jialang Lu and Xiang Chen and Jichao Wang and Dianjie Lu and Guijuan Zhang and Guangwei Gao and Zhuoran Zheng},
  title = {{DeLiVR: Differential Spatiotemporal Lie Bias for Efficient Video Deraining}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=W2eNfLmCHY},
}
Figure