Acoustic super models for large scale video event detection

Robert Mertens, Howard Lei, Luke Gottlieb, Gerald Friedland, Ajay Divakaran2011 · DOI

TL;DR

This paper introduces a novel, data-driven approach to detecting abstract video events using only audio features, bypassing the need for speech processing or manual tagging. Their "acoustic super models" achieve a 64% detection accuracy on a challenging dataset of internet videos, demonstrating the potential of audio analysis for large-scale multimedia event retrieval.

What Problem Does This Paper Address?

The internet is awash with videos, creating a pressing need for effective ways to search, cluster, and browse them. Beyond simply finding objects, there's a growing demand for detecting more abstract events like "feeding an animal" or "wedding ceremonies." Traditional methods often rely on manual tagging or domain-specific knowledge, which is time-consuming and doesn't scale well with the vast and diverse nature of online video content. This paper tackles the challenge of building robust event detectors for these abstract concepts, specifically by leveraging the audio track of videos, without requiring speech recognition or explicit human annotations for each event type.

Methods

The core of this paper's contribution lies in its data-driven approach to acoustic event detection. Instead of relying on pre-defined rules or speech analysis, they train "acoustic super models" using example query videos. These models are built using a combination of two audio feature types:

  • Linear-Frequency Cepstral Coefficients (LFCCs): These are standard features in speaker recognition, capturing the spectral envelope of sound.
  • Modulation Spectrogram (MSG) Features: This novel addition focuses on the temporal structure of the audio signal, offering a complementary representation to LFCCs.

They employ a Gaussian Mixture Model-Universal Background Model (GMM-UBM) framework, a common technique in speaker recognition. For each event category, a "super-model" is trained on all audio from its associated videos, effectively learning a generalized representation of the event's acoustic characteristics. To improve detection, they use MAP (maximum a posteriori) adaptation from a universal background GMM trained on all audio from the training set. For testing, they compute a likelihood ratio between the test audio's features and the trained event-specific GMMs. The scores from LFCC and MSG features are then combined to produce a final detection score.

Datasets

The primary dataset used is the TRECVid MED 2011 development set. This dataset consists of "found videos," meaning consumer-produced content downloaded from social networking sites. It's characterized by short, unprofessionally produced videos.

  • Training: The authors used 2040 videos across 15 event categories for training.
  • Testing: Five specific event categories were used for testing, totaling 4251 videos. The remaining videos in the test set (3755) were randomly selected and did not belong to any of the defined event categories, serving as a "catch-all" for non-target events.

This dataset differs from previous TRECVid datasets by focusing on uncurated, "wild" internet videos.

Key Insights and Ideas

  • Audio as a Powerful Signal for Abstract Events: The paper strongly suggests that audio, even without speech, contains rich information for detecting abstract video events. This challenges the common assumption that visual features are paramount for such tasks.
  • "Super Models" for Generalization: The concept of "super models" that represent an entire event kit (rather than atomic sub-events) is key to their data-driven approach, allowing for broader event detection.
  • Complementary Nature of LFCC and MSG: Combining spectral (LFCC) and temporal (MSG) audio features proved more effective than using either alone, highlighting the benefit of multi-faceted acoustic representation.
  • Machine Perception vs. Human Perception: The authors observe that machines might find different acoustic features discriminative than humans, reinforcing the value of data-driven methods that learn these machine-centric patterns.

Key Results

  • Overall Detection Accuracy: The system achieved a 64% detection accuracy on the TRECVid MED 2011 development set, which is notable as it includes videos without audio.
  • Equal Error Rate (EER): An unweighted model combination resulted in an EER of 36%. A heuristically determined weighting of LFCC and MSG features improved this to 79.4% miss probability at a 6% false alarm rate.
  • Best Performing Events: The approach performed best on the "attempting a board trick" and "working on a woodworking project" categories.
  • Computational Efficiency: Training and testing on over 6000 videos were completed in approximately 10 hours on an 8-core PC, demonstrating scalability.

Limitations and Future Work

  • Explainability: A significant limitation of data-driven approaches is the difficulty in explaining why a particular video was classified as it was. The authors acknowledge this lack of interpretability.
  • Parameter Sensitivity: The system's performance can be sensitive to parameter choices.
  • Overfitting: There's a risk of overfitting to the training data, although the use of a background model helps mitigate this.
  • Future Work:
    • Combining with Visual Features: The authors explicitly state their system is designed to be part of a multimodal framework and expect combining it with visual concept detection, OCR, and semantic analysis will yield better results.
  • Exploring Higher-Level Audio Features: Investigating more abstract audio features is a planned direction.
  • Event-Specific Classifier Performance: Addressing the fact that different classifiers perform differently on a per-event basis.
  • Improved Feature Generation: Initial experiments with adjusting LFCC frequency ranges showed promise, suggesting further tuning of feature extraction could improve performance.

Summary updated

Abstract

Given the exponential growth of videos published on the Internet, mechanisms for clustering, searching, and browsing large numbers of videos have become a major research area. More importantly, there is a demand for event detectors that go beyond the simple finding of objects but rather detect more abstract concepts, such as "feeding an animal" or a "wedding ceremony". This article presents an approach for event classification that enables searching for arbitrary events, including more abstract concepts, in found video collections based on the analysis of the audio track. The approach does not rely on speech processing, and is language-indepent, instead it generates models for a set of example query videos using a mixture of two types of audio features: Linear-Frequency Cepstral Coefficients and Modulation Spectrogram Features. This approach can be used in complement with video analysis and requires no domain specific tagging. Application of the approach to the TRECVid MED 2011 development set, which consists of more than 4000 random "wild" videos from the Internet, has shown a detection accuracy of 64% including those videos which do not contain an audio track.
BibTeX
@misc{mertens2011acoustic,
  author = {Robert Mertens and Howard Lei and Luke Gottlieb and Gerald Friedland and Ajay Divakaran},
  title = {{Acoustic super models for large scale video event detection}},
  year = {2011},
  doi = {10.1145/2072508.2072513},
}
Figure