Abstract
Shadow removal in video sequences is a challenging task due to complex lighting interactions and the need for temporal consistency. We present PSTNet, the first data-driven framework specifically designed for video shadow removal. PSTNet employs a triple-branch architecture to extract physical, spatial, and temporal features, effectively addressing the limitations of image-based methods. To bridge the gap between synthetic training data and real-world applications, we introduce a Synthetic-to-Real (S2R) strategy. Our approach outperforms existing state-of-the-art methods by 14.7% in shadow-area RMSE on a newly curated synthetic dataset, SVSRD-85.
1. Introduction
While shadow removal for static images has seen significant progress, video shadow removal remains largely unexplored. Applying image-based models to video often results in temporal flickering because these models lack an understanding of shadow dynamics and lighting persistence across frames. The primary challenges in this domain include the complexity of dynamic scenes, the difficulty of maintaining temporal coherence, and the scarcity of paired video data (shadow/shadow-free) for training deep models.
2. The PSTNet Architecture
PSTNet addresses these challenges through a specialized triple-branch architecture designed to fuse diverse characteristic features.
2.1 Physical Characteristic Branch
Instead of treating shadow removal as a black-box pixel mapping, this branch utilizes an Adaptive Exposure Estimation Module (AEEM). It models shadows based on a linear illumination equation ($L = w \times I + b$), where parameters are estimated locally to accommodate complex, non-uniform lighting across different regions of the frame.
2.2 Spatio and Temporal Branches
The Spatio Characteristic Branch operates on full-resolution images to preserve fine textures, avoiding the loss of detail common in downsampling. Simultaneously, the Temporal Characteristic Branch uses FlowNet2 to estimate optical flow between adjacent frames, enabling the model to distinguish between moving objects and moving shadows.
2.3 Feature Fusion and Attention
Features from these branches are refined by a Mask-guided Supervised Attention Module (MSAM), which directs computational focus toward shadowed regions using shadow mask supervision. Finally, a Temporal-aware Attention Block (TAB) progressively merges these multi-characteristic features to generate the final shadow-free frame.
3. Synthetic-to-Real (S2R) Strategy
To overcome the lack of real-world paired data, we utilize the SVSRD-85 dataset, created using the Grand Theft Auto V engine. To generalize to real-world footage, we implement a Synthetic-to-Real (S2R) strategy using Fourier Domain Adaptation (FDA). This technique aligns the style of real-world frames with the synthetic training distribution, allowing the model to perform effectively on real-world datasets like SBU-TimeLapse without requiring real-world paired training data.
4. Experimental Results
PSTNet demonstrates superior performance both quantitatively and qualitatively. On the SVSRD-85 dataset, it achieves a shadow-region RMSE of 12.77, significantly lower than previous leaders like DHAN (14.97) and SID (15.69).
Qualitatively, PSTNet effectively eliminates "ghost shadows"—residual artifacts often left at shadow boundaries by other methods. The model also maintains high temporal stability, producing smooth transitions in dynamic real-world scenes where frame-by-frame methods typically fail.
5. Conclusion
PSTNet represents a significant step forward in video shadow removal by integrating physical principles with deep learning. While the model shows some limitations in extremely low-contrast scenes, its ability to leverage synthetic data for real-world application through the S2R strategy provides a robust foundation for future research in video restoration and enhancement.
Summary updated