Multi-Shot Temporal Event Localization: A Benchmark

Xiaolong Liu, Yao Hu, Song Bai, Fei Ding, Xiang Bai, Philip H. S. Torr

Abstract

Current developments in temporal event or action localization usually target actions captured by a single camera. However, extensive events or actions in the wild may be captured as a sequence of shots by multiple cameras at different positions. In this paper, we propose a new and challenging task called multi-shot temporal event localization, and accordingly, collect a large-scale dataset called MUlti-Shot EventS (MUSES). MUSES has 31,477 event instances for a total of 716 video hours. The core nature of MUSES is the frequent shot cuts, for an average of 19 shots per instance and 176 shots per video, which induces large intra-instance variations. Our comprehensive evaluations show that the state-of-the-art method in temporal action localization only achieves an mAP of 13.1% at IoU=0.5. As a minor contribution, we present a simple baseline approach for handling the intra-instance variations, which reports an mAP of 18.9% on MUSES and 56.9% on THUMOS14 at IoU=0.5.

Introduction: The Challenge of Multi-Shot Localization

Most research in Temporal Action Localization (TAL) focuses on "single-shot" scenarios—videos captured by one camera without interruption, such as home videos or surveillance footage. However, the media we consume most—movies and TV shows—is professionally edited. A single "event," such as a conversation or a fight, is typically composed of many different shots from various angles, distances, and even locations.

These frequent shot cuts create a massive hurdle for AI. Standard models rely on visual consistency to track an action. In a multi-shot environment, the visual features change abruptly every few seconds. This "intra-instance variation" makes it incredibly difficult for a model to realize that a close-up of a character and a wide shot of a room are actually part of the same continuous event.

The MUSES Dataset

The core of this work is the introduction of the MUlti-Shot EventS (MUSES) dataset, designed to benchmark this complex task.

Examples of diverse shots from the MUSES dataset, highlighting the frequent shot cuts and visual variations within events.
  • Scale: 3,697 videos, 25 categories (e.g., quarrel, crying, horse riding, desk work), and 31,477 event instances.
  • Complexity: It totals 716 hours of video.
  • The "Multi-Shot" Factor: While traditional datasets like THUMOS14 or ActivityNet have very few shots per instance, MUSES averages 19 shots per event instance and 176 shots per video.

Analysis shows that shot cuts act as a form of "visual noise" that breaks feature coherence. In MUSES, the visual similarity between parts of the same event is significantly lower than in single-shot datasets.

Proposed Methodology: Temporal Aggregation

To address the noise introduced by shot cuts, we propose a baseline approach with a novel Temporal Aggregation Module. The pipeline follows a standard detection-by-classification framework but adds a specific mechanism to bridge the gap between shots:

  1. Feature Extraction: A standard I3D network extracts 1D features from video snippets.
  2. Temporal Aggregation: To handle shot boundaries, the 1D feature sequence is reshaped into a 2D grid. Applying 2D convolutions to this grid allows the model to capture both short-term dependencies (within a single shot) and long-term dependencies (across multiple shots) simultaneously.
  3. Multi-Scale Processing: A split-transform-merge strategy with different kernel sizes handles events of varying durations.
  4. Proposal Evaluation: RoI (Region of Interest) pooling and classifiers predict the event category and refine the start/end boundaries.

Key Results and Analysis

Our evaluations demonstrate that MUSES is a significantly more difficult benchmark than traditional datasets. Existing state-of-the-art (SOTA) models struggled immensely; for example, P-GCN only achieved a 13.1% mAP at IoU=0.5. Our proposed baseline raised this to 18.9% mAP.

Error analysis revealed that "Wrong Label" and "Confusion" errors are much higher on MUSES than on other datasets. This suggests that the sheer variety of visual perspectives in a multi-shot event makes it harder for the AI to understand what is happening, not just when it starts. Despite the difficulty of MUSES, our method also excelled on traditional single-shot datasets, achieving 56.9% mAP on THUMOS14, outperforming the previous SOTA by 5.3%.

Conclusion and Future Work

While our baseline provides an improvement, the 18.9% mAP on MUSES indicates the task is far from solved. Future research could explore multi-modal integration—using audio, subtitles, or facial expressions—to maintain context when the visual scene changes abruptly. This technology holds potential for automating the creation of movie trailers, highlight reels, or content summaries by accurately identifying key plot points across edited footage.

The dataset and project code are available at https://songbai.site/muses/.

Summary updated

Abstract

Current developments in temporal event or action localization usually target actions captured by a single camera. However, extensive events or actions in the wild may be captured as a sequence of shots by multiple cameras at different positions. In this paper, we propose a new and challenging task called multi-shot temporal event localization, and accordingly, collect a large-scale dataset called MUlti-Shot EventS (MUSES). MUSES has 31,477 event instances for a total of 716 video hours. The core nature of MUSES is the frequent shot cuts, for an average of 19 shots per instance and 176 shots per video, which induces large intra-instance variations. Our comprehensive evaluations show that the state-of-the-art method in temporal action localization only achieves an mAP of 13.1% at IoU=0.5. As a minor contribution, we present a simple baseline approach for handling the intra-instance variations, which reports an mAP of 18.9% on MUSES and 56.9% on THUMOS14 at IoU=0.5. To facilitate research in this direction, we release the dataset and the project code at https://songbai.site/muses/.
BibTeX
@inproceedings{liu2021multi,
  author = {Xiaolong Liu and Yao Hu and Song Bai and Fei Ding and Xiang Bai and Philip H. S. Torr},
  title = {{Multi-Shot Temporal Event Localization: A Benchmark}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=8K17jS8-V2},
}
Figure