TL;DR
PSTNet is the first data-driven framework specifically designed for video shadow removal. By integrating physical lighting models, spatial texture preservation, and temporal motion cues, it outperforms existing state-of-the-art methods by 14.7% in shadow-area RMSE and introduces a novel synthetic-to-real (S2R) strategy to bridge the gap between game-engine training data and real-world footage.
What Problem Does This Paper Address?
Shadow removal is a classic computer vision challenge, but most research has focused on static images. Applying image-based shadow removers to video often results in "flickering" or temporal instability because these models don't understand how shadows move or how lighting persists across frames.
The authors identify three major hurdles:
- Complexity: Shadows in dynamic scenes involve complex interactions between moving light sources, varying textures, and occluding objects.
- Temporal Coherence: Simply processing frames individually ignores the valuable information contained in adjacent frames.
- Data Scarcity: Training a deep learning model requires "paired" data (the same video with and without shadows). In the real world, it is nearly impossible to record a dynamic scene twice with identical motion but different lighting.
Methods
The authors propose PSTNet, a triple-branch architecture that extracts and fuses three distinct types of features:
- Physical Characteristic Branch: Instead of treating shadow removal as a simple pixel-mapping task, this branch uses an Adaptive Exposure Estimation Module (AEEM). It models shadows as a linear illumination problem ($L = w \times I + b$) but estimates these parameters locally for different regions to handle complex lighting.
- Spatio Characteristic Branch: This branch operates on the original image resolution (without downsampling) to ensure that fine textures are preserved in the final output.
- Temporal Characteristic Branch: Using FlowNet2, the model calculates optical flow between adjacent frames. This helps the network distinguish between moving objects and moving shadows.
To refine these features, the authors include a Mask-guided Supervised Attention Module (MSAM), which uses a shadow mask to "tell" the network to focus its computational power on the shadowed regions. Finally, a Temporal-aware Attention Block (TAB) progressively merges these features to produce the final shadow-free frame.
To handle the transition from synthetic training data to real-world testing, they introduce S2R (Synthetic-to-Real). This uses Fourier Domain Adaptation (FDA) to swap the low-frequency "style" components of real-world frames with synthetic ones, allowing the model to work on real footage without needing real-world paired training data.
Datasets
- SVSRD-85 (New): A synthetic dataset created using the game Grand Theft Auto V (GTAV). By controlling the game's shadow renderer, the authors collected 85 videos (4,250 frames) of perfectly paired shadow/shadow-free sequences with dynamic motion.
- SBU-TimeLapse: A real-world dataset used to verify the model’s generalization and the effectiveness of the S2R strategy.
Key Insights and Ideas
- Physics-Informed Deep Learning: The paper demonstrates that pure "black box" CNNs struggle with shadow removal. By forcing the network to learn physical parameters (exposure and offset), the model is better constrained and produces more realistic results.
- Local vs. Global Adaptation: Previous physical models used a single set of parameters for an entire image. PSTNet’s AEEM proves that estimating exposure parameters per region is essential for scenes with diverse textures.
- The Power of Synthetic Data: The success of SVSRD-85 suggests that high-fidelity game engines are a viable proxy for training video restoration models, provided a domain adaptation step (like S2R) is used.
Key Results
- Quantitative: On the SVSRD-85 dataset, PSTNet achieved an RMSE of 12.77 in shadow regions, a significant improvement over the previous best-performing model, DHAN (14.97), and the physical-based SID (15.69).
- Qualitative: Visual comparisons show that PSTNet is much better at removing "ghost shadows"—the faint, blurry outlines often left behind by other models at the edges of the original shadow.
- Efficiency: The S2R strategy allowed the model to generalize to real-world SBU-TimeLapse videos effectively, reducing artifacts and maintaining temporal smoothness where image-based methods failed.
Limitations and Future Work
The authors acknowledge a few key limitations:
- Low Contrast Scenes: The model struggles when there is very low illumination contrast (i.e., when the shadow is nearly as dark as the surrounding non-shadow area).
- Domain Bias: While S2R helps, there is still a visible gap between synthetic and real-world performance. The authors suggest that future work should focus on further reducing this domain bias to make synthetic-driven models even more robust for "in-the-wild" video.
Summary updated