Key Event Detection in Video using ASR and Visual Data

Niraj Shrestha, Aparna Nurani Venkitasubramanian, Marie-Francine MoensVL @ COLING 2014 · OpenReview

TL;DR

This paper tackles the challenge of automatically identifying key events in videos, specifically royal weddings, by combining Automatic Speech Recognition (ASR) with visual data. The authors propose a novel approach that leverages external knowledge sources like Wikipedia to learn event patterns and then applies this to noisy ASR transcripts, showing promising results with only a slight degradation compared to cleaner subtitle data.

What Problem Does This Paper Address?

The explosion of multimedia data necessitates efficient ways to index and retrieve information. Manually annotating videos is impractical, making automatic key event detection crucial for effective search and mining. While much research focuses on visual analysis or clean textual data like manual transcripts, this paper addresses the gap of using readily available but inherently noisy ASR data for event detection. Specifically, it aims to identify significant moments in royal wedding videos and the people involved, a task complicated by the inaccuracies of ASR and the presence of non-event-related speech (e.g., anchor commentary).

Methods

The core methodology involves a two-pronged approach:

  1. Acquiring Background Knowledge: The system first learns typical wedding events by extracting information from Wikipedia articles related to royal weddings. This "genre-specific corpus" is then weighted against a "generic corpus" (all English Wikipedia articles) to prioritize more relevant and less common wedding events. A state-of-the-art event annotator is used to extract events and their arguments from text.
  2. Event Identification and Person Alignment:
    • Event Detection: The learned event keywords are then scanned for in the ASR (or subtitle) data. When a keyword is found, the associated frames are considered to contain that event.
  3. Name and Face Alignment: To identify persons involved, the system uses a gazetteer of known names (specifically for brides and bridegrooms in this preliminary work). Faces are detected in video key frames using a face detector and then clustered. An Expectation-Maximization (EM) algorithm is employed to align detected faces with names found in the co-occurring textual data (ASR or subtitles) within a time window.

Datasets

  • Video Data: The DVD "Britain's Royal Weddings" published by the BBC, with a duration of 116 minutes.
  • Ground Truth: Manually annotated events and their start/end times by two annotators.
  • External Corpora: Wikipedia articles related to British royal weddings and general English Wikipedia articles for knowledge acquisition.
  • ASR Data: Generated using the FBK (2013) system from the video's audio.
  • Subtitle Data: Provided with the DVD, segmented into approximately 3-second intervals.

Key Insights and Ideas

  • Leveraging Noisy ASR: The paper demonstrates that even with the inherent inaccuracies of ASR, it can still be a viable source for key event detection, offering a practical alternative to manually transcribed data.
  • External Knowledge for Event Learning: Using Wikipedia to learn event patterns is an effective way to bootstrap the event detection process for a specific domain like royal weddings, reducing the need for extensive manual annotation of the target videos.
  • Challenges of Anchor Commentary: The presence of anchors or interviewers talking about events without the actual event being visually present in the frame poses a significant challenge for name-face alignment, as it can lead to misattributions.
  • EM for Alignment Refinement: The EM algorithm proves useful in refining the alignment between detected faces and names, helping to mitigate some of the ambiguity introduced by noisy data and non-event-related speech.

Key Results

  • Event Identification Accuracy: The system successfully identified learned events in both subtitle and ASR data, with only a "slightly higher" error rate when using ASR compared to subtitles.
  • Name and Face Alignment: Table 1 shows precision (P), recall (R), and F1 scores for name-face alignment. On subtitles, P, R, and F1 were 38.095, 21.622, and 27.586 respectively. For ASR, these scores were 36.585, 17.857, and 24. While not perfect, these results indicate some success in aligning names with faces.
  • Event Segmentation: Table 2 shows a WinDiff score of 11.06 for subtitles and 13.80 for ASR, indicating a slight degradation in event segmentation accuracy when using ASR data, attributed to ASR errors.

Limitations and Future Work

  • ASR Errors: The primary limitation acknowledged is the noisy and inaccurate nature of ASR data, which impacts both event detection and name-face alignment.
  • Non-Event Speech: The presence of interviews and anchor commentary that discuss events without showing them complicates name-face alignment.
  • Limited Person Scope: In this preliminary work, the focus was on brides and bridegrooms. Future work could expand to identify all mentioned persons.
  • Improving External Learning: The authors are interested in improving the process of learning events from external corpora.
  • Handling Interview Content: A key area for future improvement is developing methods to better handle interview content and improve name-face alignment in such scenarios.
  • Event Boundary Detection: Finding precise event boundaries in frame sequences corresponding to textual mentions remains a challenge.

Summary updated

BibTeX
@inproceedings{shrestha2014key,
  author = {Niraj Shrestha and Aparna Nurani Venkitasubramanian and Marie-Francine Moens},
  title = {{Key Event Detection in Video using ASR and Visual Data}},
  booktitle = {VL @ COLING},
  year = {2014},
  url = {https://openreview.net/forum?id=ImPiPGVsoI},
}
Figure