Two Stage Shot Boundary Detection via Feature Fusion and Spatial-Temporal Convolutional Neural Networks

Lifang Wu, Shuai Zhang, Meng Jian, Zhe Lu, Dong WangIEEE Access 2019 · OpenReview

Abstract

Shot boundary detection is essentially to detect the position of frames where the shot changes. It has been actively studied in video analysis and management for convenience, which becomes a key technique with the rapid proliferation of rich and diverse videos. With respect to the complex characteristics of different shots in varying length and content variation property, in this paper we present a two stage method for shot boundary detection (TSSBD) which distinguishes abrupt shot by fusing color histogram and deep features, and locate gradual shot changes with C3D-based deep analysis. Abrupt shot changes are detected firstly as it occurs between two frames, which divides the complete video into segments containing gradual transitions; Over these video segments, gradual shot change detection is implemented using 3D-convolutional neural network, which classifies clips into specific gradual shot change types; Finally, an effective merging strategy is proposed to locate positions of gradual shot transitions. The experimental analysis illustrates that the proposed progressive method is capable of detecting both abrupt shot transitions and gradual shot transitions accurately.

What Problem Does This Paper Address?

In the era of massive video databases, Shot Boundary Detection (SBD) is the essential first step for video retrieval, summarization, and semantic analysis. However, achieving high accuracy is challenging due to the variety of transition types and potential "noise" like camera motion or lighting changes.

Shot transitions generally fall into two categories: abrupt (hard cuts) and gradual (dissolves, fades, and wipes). Abrupt cuts occur instantly between two frames but can be confused with camera flashes or fast-moving objects. Gradual transitions are more complex as they span multiple frames, often mimicking simple camera motion or illumination shifts to traditional algorithms.

Examples of different shot transition types, including abrupt cuts and gradual transitions like dissolves and fades.

Methods: The TSSBD Framework

The researchers propose a Two-Stage Shot Boundary Detection (TSSBD) framework that employs a "divide and conquer" strategy to handle the distinct characteristics of different transitions.

The Two-Stage Shot Boundary Detection (TSSBD) framework, showing abrupt shot detection followed by sliding window analysis for gradual transitions.

Stage 1: Abrupt Shot Detection

The first stage acts as a filter to identify hard cuts and segment the video. To ensure robustness, the system fuses two distinct feature types:

  • Blocked Color Histograms: These capture spatial color distribution at a pixel level.
  • Deep Features: High-level semantic information is extracted using the Fc6 layer of an AlexNet.

By combining these, the model can distinguish between actual shot changes and semantic-preserving changes like motion. Dedicated modules for Flash Detection and Large Motion Detection further reduce false positives.

Stage 2: Gradual Shot Detection

Once the video is segmented, the system focuses on the more complex gradual transitions within those segments using a C3D (3D Convolutional Neural Network). Unlike 2D CNNs, C3D uses 3D filters to capture spatial-temporal features, allowing the model to "see" how pixel patterns evolve over time.

  • Sliding Window: A 16-frame window with 87.5% overlap is used to scan segments.
  • Majority Voting & Gap Filling: A voting strategy assigns labels to frames based on overlapping window predictions. A "Gap Filling" algorithm then merges fragmented predictions to ensure a single, cohesive transition is identified.

Key Insights and Results

The study highlights that Feature Fusion is critical for abrupt cuts; color histograms and deep features are complementary rather than redundant. Furthermore, the use of Temporal Context via C3D is far more effective for gradual transitions than frame-by-frame comparison.

The TSSBD method was evaluated on the TRECVID 2005 benchmark dataset. Key results include:

  • Abrupt Detection: Achieved an F-score of 0.935, outperforming standalone C3D or histogram methods.
  • Overall Performance: TSSBD reached an F-score of 0.894, significantly outperforming other deep learning models like CNN+LSTM (0.708) and DeepSBD (0.844), as well as traditional TRECVID contest leaders (0.786).

Conclusion

TSSBD provides a robust solution for video segmentation by effectively handling both simple cuts and complex gradual transitions. While the C3D-based sliding window approach is computationally intensive, its superior accuracy sets a new benchmark for the field. Future work may focus on optimizing inference speed for real-time industrial applications.

Summary updated

Abstract

Shot boundary detection is essentially to detect the position of frames where the shot changes. It has been actively studied in video analysis and management for convenience, which becomes a key technique with the rapid proliferation of rich and diverse videos. With respect to the complex characteristics of different shots in varying length and content variation property, in this paper we present a two stage method for shot boundary detection (TSSBD) which distinguishes abrupt shot by fusing color histogram and deep features, and locate gradual shot changes with C3D-based deep analysis. Abrupt shot changes are detected firstly as it occurs between two frames, which divides the complete video into segments containing gradual transitions; Over these video segments, gradual shot change detection is implemented using 3D-convolutional neural network, which classifies clips into specific gradual shot change types; Finally, an effective merging strategy is proposed to locate positions of gradual shot transitions. The experimental analysis illustrates that the proposed progressive method is capable of detecting both abrupt shot transitions and gradual shot transitions accurately.
BibTeX
@inproceedings{wu2019two,
  author = {Lifang Wu and Shuai Zhang and Meng Jian and Zhe Lu and Dong Wang},
  title = {{Two Stage Shot Boundary Detection via Feature Fusion and Spatial-Temporal Convolutional Neural Networks}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=rNV0wwwFLE},
}
Figure