Structure-Aware Spatial-Temporal Interaction Network for Video Shadow Detection

Housheng Wei, Guanyu Xing, Jingwei Liao, Yanci Zhang, Yanli LiuIJCAI 2024

Abstract

Video shadow detection is a critical yet challenging task in computer vision, requiring both high-precision spatial localization and robust temporal consistency. We present the Structure-Aware Spatial-Temporal Interaction Network (SSTINet), a novel framework designed to address the limitations of existing methods, such as ambiguous semantics and temporal flickering. By decoupling shadow prediction into "body" and "detail" components and utilizing a hybrid Transformer-CNN architecture with bidirectional feature sharing, SSTINet achieves a significant 41.3% reduction in Mean Absolute Error (MAE) over previous state-of-the-art models on the Visha dataset.

Introduction

Detecting shadows in video sequences is significantly more complex than in static images. Beyond the basic requirement of pixel-level accuracy, video models must maintain temporal consistency to ensure shadows do not flicker or change shape erratically between frames. Current research faces three primary hurdles:

  1. Ambiguous Semantics: Distinguishing between dark objects (e.g., a black vehicle) and shadows cast on dark surfaces remains difficult.
  2. Fine Detail Loss: Many models fail to capture the soft, blurry edges (penumbra) of shadows, resulting in "hollow" or jagged segmentation masks.
  3. Temporal Instability: Processing frames in isolation often leads to shadows popping in and out of existence across a sequence.

SSTINet addresses these issues by modeling the "anatomy" of a shadow—the structural relationship between its core and its boundary—while ensuring intelligent feature sharing across time.

Methodology

The SSTINet architecture employs a "Siamese" configuration, where the current frame ($I_t$) and the previous frame ($I_{t-1}$) are processed through parallel, interacting branches.

The SSTINet architecture utilizes a Siamese setup with hybrid feature extraction and interaction modules to maintain spatial detail and temporal consistency.

Key Components

  • Hybrid Backbone: The model uses a Transformer to capture global context (essential for understanding light sources) and a lightweight CNN to capture local details and sharp edges.
  • Spatial Feature Injection Module (SFIM): This module "injects" local details from the CNN into the Transformer’s attention blocks, preventing the loss of fine textures during global processing.
  • Temporal Feature Interaction Module (TFIM): This enables a bidirectional exchange of features between adjacent frames, allowing the current frame to "borrow" information from the past to stabilize the shadow's shape.
  • Structure-Aware Shadow Prediction Module (SASPM): Rather than simple binary classification, this module uses Distance Transformation to learn the shadow "body" (center) and "detail" (edges) separately before fusing them into a final mask.

Experiments and Results

The model was rigorously evaluated on the Visha dataset, which consists of 120 scenes and approximately 11,805 images. Performance was measured using standard metrics including Mean Absolute Error (MAE), Intersection over Union (IoU), and a specialized Consistency (TS) metric.

Qualitative comparison showing SSTINet's ability to produce cleaner, more consistent shadow masks than previous methods across various challenging scenes.

Key Findings

  • Superior Accuracy: SSTINet achieved a MAE of 0.017, representing a 41.3% error reduction compared to the previous best model, Scotch (0.029).
  • Enhanced Stability: The model demonstrated significantly higher temporal stability, with a TS score of 0.853 compared to 0.640 for Scotch.
  • Practical Application: In Augmented Reality (AR) tests, SSTINet's precise shadow masks allowed virtual objects to blend seamlessly into real-world environments, avoiding the "floating" effect common with less accurate models.

Conclusion

SSTINet advances video shadow detection by recognizing that shadows are structured entities with distinct bodies and boundaries. By combining global transformer views with local CNN details and enforcing temporal interaction, the model sets a new benchmark for accuracy and stability. Future work will focus on reducing computational complexity for real-time applications and implementing long-term memory banks to handle extended occlusions.

Summary updated

BibTeX
@inproceedings{wei2024structure,
  author = {Housheng Wei and Guanyu Xing and Jingwei Liao and Yanci Zhang and Yanli Liu},
  title = {{Structure-Aware Spatial-Temporal Interaction Network for Video Shadow Detection}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure