Towards Global Video Scene Segmentation with Context-Aware Transformer

Yang Yang, Yurui Huang, Weili Guo, Baohua Xu, Dingyin XiaAAAI 2023 · DOI

Abstract

Video scene segmentation is a critical task for long-form video understanding, yet it faces significant challenges due to coarse-grained labels, high-order narrative coherence, and data scarcity. We present the Context-Aware Transformer (CAT), a framework that leverages a local-global attention mechanism and self-supervised pre-training to achieve state-of-the-art performance. By pre-training on unlabeled movies with four specialized pretext tasks, CAT learns semantic structures that generalize effectively to scene boundary detection. Our approach outperforms existing methods on the MovieNet benchmark, achieving a new state-of-the-art Average Precision (AP) of 59.55%.

The Challenge of Video Scene Segmentation

In video analysis, a fundamental distinction exists between a shot (a continuous camera take) and a scene (a series of shots forming a semantically cohesive story). While detecting shot changes is relatively straightforward based on visual discontinuities, segmenting scenes is complex due to three primary hurdles:

  1. Coarse-grained labels: Most datasets provide only binary boundary labels, offering no insight into the underlying narrative content.
  2. High-order coherence: Scenes are defined by story arcs rather than simple visual similarity. Shots in different locations may belong to the same scene, while visually similar shots in the same room might belong to different narrative segments.
  3. Data Scarcity: Manual annotation of scene boundaries in feature-length films is prohibitively expensive, leading to a lack of large-scale labeled training data.

The Context-Aware Transformer (CAT) Architecture

To address these challenges, the CAT framework utilizes a dual-encoding approach designed to capture both immediate and long-term dependencies within a video sequence.

The architecture of the Context-Aware Transformer (CAT), featuring a dual-encoder design for local and global context.

As shown in Figure 1, the model consists of two primary components:

  • Local Encoder: This component focuses on a fixed window of neighboring shots to capture immediate intra-scene dependencies and local flow.
  • Global Encoder: This component analyzes the entire video sequence to identify long-term correlations and inter-scene transitions, which are essential for recognizing when a narrative arc has concluded.

Self-Supervised Pre-training

To overcome the scarcity of labeled data, CAT is pre-trained on unlabeled videos using four self-supervised "pretext" tasks:

  • Shot Masked Modeling (SMM): Predicting missing shot features based on surrounding context.
  • Shot Order Modeling (SOM): Reconstructing the correct sequence from shuffled shots.
  • Global Scene Matching (GSM): Using "pseudo-boundaries" (derived from visual similarity thresholds) to teach the model to cluster semantically related shots.
  • Local Scene Matching (LSM): Distinguishing whether adjacent shots belong to the same semantic unit.

This pre-training phase allows the model to learn the underlying structure of film before being fine-tuned on a smaller set of human-annotated boundary labels.

Experimental Results

CAT was evaluated primarily on the MovieNet benchmark, which includes 1,100 movies. The model set a new high bar for performance, significantly outperforming previous methods like BaSSL.

Performance comparison of CAT against state-of-the-art methods on the MovieNet benchmark.

As detailed in Table 1, CAT achieved an AP of 59.55%, representing a 2.15% improvement over the previous state-of-the-art. The results demonstrate that the combination of local-global attention and self-supervised pre-training is highly effective. Notably, even when trained on a reduced dataset (the "Train only" split), CAT remained competitive, highlighting the robustness of its learned features.

Beyond visual data, the researchers found that incorporating audio features via late fusion further boosted performance to 60.20% AP, confirming that auditory cues like background music and ambient noise provide vital context for scene transitions.

Conclusion and Future Work

The Context-Aware Transformer demonstrates that global context is mandatory for accurate scene segmentation. While the current model achieves state-of-the-art results, future work will explore more sophisticated multi-modal fusion strategies, such as cross-modal attention, to better integrate audio and visual streams. Additionally, moving toward an end-to-end system that processes raw video frames directly could capture even finer narrative nuances.

Summary updated

BibTeX
@inproceedings{yang2023towards,
  author = {Yang Yang and Yurui Huang and Weili Guo and Baohua Xu and Dingyin Xia},
  title = {{Towards Global Video Scene Segmentation with Context-Aware Transformer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25426},
}
Figure