Multi-shot Temporal Event Localization: a Benchmark

Xiaolong Liu, Yao Hu, Song Bai, Fei Ding, Xiang Bai, Philip H. S. Torr

Temporal Action Localization (TAL)—the task of identifying "what" happened and "when" it started and ended in a video—has seen massive strides recently. However, most research has focused on "single-shot" scenarios: videos captured by one camera without interruption, typical of user-generated content or sports. Real-world professional media, like movies and TV shows, are a different beast entirely. They rely on professional editing, where a single event (like a conversation or a chase) is fragmented across dozens of camera angles and shot cuts.

In their paper, "Multi-shot Temporal Event Localization: a Benchmark," researchers from HUST, Alibaba, and Oxford introduce a new dataset and a specialized approach to tackle the visual chaos of edited video.

The Challenge of Intra-Instance Variation

The core challenge addressed is intra-instance variation. In a typical TAL dataset like THUMOS14, an action usually looks visually consistent from start to finish. In professional media, a single semantic event might cut between vastly different visual environments.

Figure 4: Examples of diverse camera shots (a-d) that may constitute a single multi-shot event in professional media.

To a computer vision model, these shot cuts look like total scene changes. The background, lighting, and even the actors present can change in a single frame. Existing models often fail to realize these disparate shots belong to the same semantic event. The authors argue that we need a benchmark that reflects the "multi-shot" nature of the media we consume most.

The MUSES Dataset

To address this gap, the authors propose MUSES (MUlti-Shot EventS), a large-scale dataset specifically designed for localizing events that span multiple camera shots.

  • Scale: 31,477 event instances across 716 hours of video.
  • Complexity: 25 categories (e.g., quarrel, horse riding, desk work).
  • Shot Density: An average of 19 shots per instance and 176 shots per video.

Analysis of MUSES reveals a significant "Self-Similarity Gap." The cosine similarity between snippets of the same event is significantly lower than in other datasets, proving that shot cuts create massive internal visual divergence.

Proposed Methodology: Temporal Aggregation

The authors propose a Temporal Aggregation Module to mitigate the noise introduced by shot cuts and better "stitch" features across boundaries.

  1. Feature Extraction: A standard I3D backbone extracts 1D temporal features.
  2. 2D Reshaping: To capture both local and long-range dependencies, the 1D feature sequence is reshaped into a 2D map ($H \times W$).
  3. 2D Convolutional Aggregation: By applying 2D convolutions to this reshaped map, the model can "see" across multiple temporal units simultaneously. This helps the model learn feature coherence, effectively smoothing out the "shocks" to the feature space caused by shot cuts.
  4. Multi-scale Branching: Since events vary in length, a multi-branch architecture with different kernel sizes captures information at various temporal scales.

Key Results and Insights

The researchers found that existing state-of-the-art models struggle significantly with MUSES. For instance, the P-GCN model only achieved a 13.1% mAP (mean Average Precision) at IoU=0.5, highlighting the increased difficulty compared to traditional TAL.

The proposed temporal aggregation baseline reached 18.9% mAP on MUSES—a significant jump over existing methods. Furthermore, when applied to the older THUMOS14 dataset, the baseline achieved 56.9% mAP, setting a new high bar at the time of publication and proving that handling temporal variations helps even in single-shot scenarios.

Error analysis revealed that classification is the primary culprit for low performance. Frequent cuts confuse the classifier, leading to "Wrong Label" and "Confusion" errors rather than simple boundary localization errors. Interestingly, the model performed better on "large" instances (those with more than 20 shots) than on "small" ones, suggesting that more shots provide more total context for the model to eventually resolve the event type.

Conclusion and Future Work

While the proposed baseline is a step forward, an 18.9% mAP indicates that multi-shot localization remains an open challenge. Future research may need to incorporate multi-modality (audio, subtitles) or complex relational reasoning to understand the narrative link between disparate camera angles. MUSES provides the necessary benchmark to drive these innovations in professional media analysis.

Summary updated

BibTeX
@inproceedings{liu2021multi,
  author = {Xiaolong Liu and Yao Hu and Song Bai and Fei Ding and Xiang Bai and Philip H. S. Torr},
  title = {{Multi-shot Temporal Event Localization: a Benchmark}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=8K17jS8-V2},
}
Figure