Robust event detection from spoken content in consumer domain videos

Stavros Tsakalidis, Xiaodan Zhuang, Roger Hsiao, Shuang Wu, Pradeep Natarajan, Rohit Prasad, Prem NatarajanInterspeech 2012

TL;DR

This paper tackles the challenge of automatically detecting events in consumer-generated videos (like YouTube clips) by focusing on the spoken audio. They introduce a robust, data-driven approach that improves keyword selection, expands keyword vocabulary using word clusters, and enhances keyword spotting in noisy conditions, leading to significantly better event detection performance.

What Problem Does This Paper Address?

The explosion of user-generated content on platforms like YouTube presents a massive challenge for content analysis and retrieval. While professional content like broadcast news is often well-labeled, consumer videos are far more heterogeneous, with noisy audio, sporadic speech, and a lack of manual annotations. This makes it difficult to automatically identify specific events within these videos. Traditional methods often rely on manually labeled data or highly structured audio, which are unsuitable for this domain. This paper aims to bridge this gap by developing robust methods to extract meaningful information from the spoken content of consumer videos, even under challenging conditions.

Methods

The authors propose an integrated approach that enhances the audio analysis component of the BBN VISER system. Their key innovations include:

  • Robust Data-Driven Keyword Selection: Instead of relying solely on raw ASR output, they incorporate word-level confidence measures. This modified TF-IDF metric helps to down-weight unreliable transcriptions, making keyword selection more robust to ASR errors.
  • Automatic Discovery of Word-Classes for Keyword Expansion: To combat the sparsity of keywords in consumer videos, they automatically discover semantically related word-classes. They leverage an external thesaurus for pairwise word similarities and then use affinity propagation clustering to group words into meaningful clusters. This allows the system to recognize events even if the exact keywords aren't present in the transcript.
  • Keyword Spotting for Improved Recall in Noisy Conditions: They move beyond a simple "1-best" ASR output and utilize confusion networks (lattices) from the ASR system. This provides a richer set of potential keywords. To address the issue of false alarms in these lattices, they propose a score mapping technique. This technique learns a mapping from the posterior probability of a word in the lattice to its actual probability of being correct, effectively filtering out unreliable hits.

Datasets

The primary dataset used for evaluation is the Multimedia Event Detection (MED) task dataset from TRECVID 2011.

  • The authors selected a subset of videos containing at least some speech for their experiments.
  • The training set consists of 4852 videos, with 1200 videos from 6 specific events of interest and the rest from a background class.
  • The test set comprises 2218 videos, with 410 videos from the 6 events of interest.
  • A small set of 100 video clips with annotated speech transcriptions was used for developing and evaluating the score mapping technique.

Key Insights and Ideas

  • Confidence is Key: Simply using the most likely transcription from an ASR system is insufficient for noisy, consumer-grade audio. Incorporating confidence scores significantly improves the reliability of keyword selection.
  • Beyond Exact Matches: Relying on exact keyword matches is problematic due to the variability and sparsity of speech in consumer videos. Automatically discovering semantically related word clusters is a powerful way to expand the system's understanding and improve recall.
  • Lattices Offer More, But Need Careful Handling: ASR lattices contain more information than the single best hypothesis, but they also introduce more noise. A learned mapping from posterior probabilities to actual correctness (P(corr)) is crucial for effectively leveraging this richer information.
  • Integrated Approach Yields Best Results: The combination of robust keyword selection, keyword expansion via word clusters, and confusion network-based keyword spotting leads to the most significant improvements, demonstrating the synergy of these techniques.

Key Results

The proposed integrated approach achieved a 26% relative improvement in event detection performance compared to the baseline ASR-based features used in the BBN VISER system for the 2011 TRECVID MED evaluation.

  • Using word confidences for both keyword selection and extraction yielded a 10.4% relative improvement.
  • Leveraging confusion networks (beyond the 1-best hypothesis) resulted in a 12.4% relative improvement.
  • Fusing confusion network features with word-class features provided an additional 1.6% relative improvement.
  • The Word Error Rate (WER) for the adapted ASR system was reduced to 35.8% from a baseline of 48.2%.

Limitations and Future Work

  • Limited Event Scope: The experiments were conducted on a specific set of 6 events. The generalizability of the approach to a much wider range of events, especially those with less distinct spoken content, remains to be fully explored.
  • Dependence on ASR Quality: While the methods aim to be robust to ASR errors, the performance is still inherently tied to the quality of the initial speech recognition.
  • Future Work: The authors suggest extending the approach to consider multi-word keywords and further exploring the fusion of audio and visual features for even more comprehensive event detection. The potential for unsupervised learning of word-classes and the application of these techniques to other multimedia analysis tasks are also implied.

Summary updated

BibTeX
@inproceedings{tsakalidis2012robust,
  author = {Stavros Tsakalidis and Xiaodan Zhuang and Roger Hsiao and Shuang Wu and Pradeep Natarajan and Rohit Prasad and Prem Natarajan},
  title = {{Robust event detection from spoken content in consumer domain videos}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure