Modality-Aware Shot Relating and Comparing for Video Scene Detection

Jiawei Tan, Hongxing Wang, Kang Dang, Jiaxin Li, Zhilong OuAAAI 2025 · DOI

TL;DR

This paper introduces MASRC, a novel framework for video scene detection that leverages both visual entity and place information to better understand shot relationships. By treating these modalities separately and using graph-based methods, MASRC captures both long-range and short-range dependencies, leading to more accurate scene boundary identification.

What Problem Does This Paper Address?

Video scene detection is crucial for organizing and understanding video content, enabling applications like video retrieval and storyline construction. The core challenge lies in accurately identifying where one scene ends and another begins. This requires understanding the context of a shot, which involves analyzing its surrounding shots. Existing methods often treat different types of visual information (like objects and locations) similarly, failing to capture the distinct ways they contribute to scene transitions. For instance, a recurring character (entity) might span multiple scenes, while a consistent location might define a single scene. This paper addresses this by proposing a "modality-aware" approach that specifically models the unique temporal relationships of entities and places.

Methods

The proposed method, Modality-Aware Shot Relating and Comparing (MASRC), consists of two main components:

  • Modality-Aware Shot Relating (MASR): This component captures temporal relationships between shots by treating visual entity and place modalities distinctly.
  • Entity-based Long-Term Dependency: An "Entity Jumping Graph" (EJG) is constructed to link shots with similar visual entities, even if they are temporally distant. This captures long-range dependencies, such as the reappearance of the same character. Graph Convolutional Networks (GCNs) are then used on this graph to extract "long-range affinity features."
    • Place-based Short-Term Dependency: A "Place Continuity Graph" (PCG) is used to connect consecutive shots that depict the same place, capturing short-range dependencies. This involves identifying "wide shots" (overview of a place) and "detail shots" (zooming into specific aspects of a place) and modeling their affiliation. GCNs are applied here to obtain "short-range affinity features."
  • Multi-shots Comparison Detection (MCD): After obtaining enriched shot features from MASR, this component focuses on identifying scene ending shots. It compares the semantic similarity between the "fore-and-aft" contexts of a target shot using a similarity matrix. This matrix is then processed by a Convolutional Neural Network (CNN) to generate a "relation vector." Finally, a Multi-Layer Perceptron (MLP) predicts the probability of the shot being an ending shot.

Datasets

The paper evaluates MASRC on three widely used video scene detection datasets:

  • MovieNet: A large dataset with annotated scene boundaries (MovieScenes subset).
  • BBC: Episodes from the BBC educational TV series "Planet Earth."
  • OVSD: A dataset extracted from movie scripts.

The authors also mention using unlabeled videos from MovieNet for self-supervised pre-training.

Key Insights and Ideas

  • Modality-Specific Temporal Relations: The core insight is that entities and places contribute differently to scene structure. Entities are better for long-range dependencies (e.g., character arcs), while places are better for short-range dependencies (e.g., defining a scene's location).
  • Wide vs. Detail Shots: Differentiating between "wide shots" (scene overview) and "detail shots" (specific elements within a scene) is crucial for effectively modeling place-based relationships.
  • Contextual Comparison for Scene Endings: Simply looking at a shot's features isn't enough to detect scene endings. Comparing the semantic consistency before and after a shot is a more robust indicator of a scene transition.
  • Complementarity of Modalities: Combining both entity and place information consistently leads to the best performance, highlighting their complementary roles in understanding video scenes.

Key Results

  • MASRC significantly outperforms state-of-the-art methods on the MovieNet dataset across various training settings (fully supervised, self-supervised, and self-supervised transfer learning), achieving the highest AP, mIoU, and F1 scores.
  • The proposed MASRC achieves superior performance compared to other methods when using the same ending shot detectors, demonstrating the effectiveness of its MASR component.
  • The MCD component also shows improvements when combined with existing temporal relation modeling methods, highlighting its versatility.
  • Ablation studies confirm the importance of each component: ELD, PSD, and MCD all contribute to performance gains, with the combination of ELD and PSD yielding the best results.
  • The method shows strong generalization capabilities, achieving competitive results on the OVSD and BBC datasets even without fine-tuning after training on MovieNet.

Limitations and Future Work

  • Wide Shot Identification: The paper acknowledges that their method for identifying "wide shots" based on similarity might differ from reality due to the inability to access focal length information.
  • Hyperparameter Sensitivity: While optimal hyperparameters (temporal window scale T=14, entity feature neighbor scale k=4) are identified, future work could explore more adaptive or learned approaches to these parameters.
  • Future Work: The authors suggest that their MASRC framework can be extended to other video understanding tasks that rely on understanding temporal relationships between shots. The plug-and-play nature of MASR and MCD also suggests their applicability to various downstream tasks.

Summary updated

BibTeX
@inproceedings{tan2025modality,
  author = {Jiawei Tan and Hongxing Wang and Kang Dang and Jiaxin Li and Zhilong Ou},
  title = {{Modality-Aware Shot Relating and Comparing for Video Scene Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32773},
}
Figure