TL;DR
This paper introduces a novel audio-only approach for classifying video genres, combining low-level acoustic features with higher-level semantic ones like speaker interactivity and speech quality. Their method achieves a remarkable 91% identification rate on a diverse corpus, outperforming previous audio-based techniques and rivaling audio-visual methods.
What Problem Does This Paper Address?
The explosion of online video content necessitates efficient ways to organize and search these vast collections. While visual analysis has been a primary focus, this paper argues for the robustness and effectiveness of audio-only analysis, especially in the face of variable video quality and transcription challenges. Specifically, they aim to classify videos into genres like news, movies, cartoons, music, and commercials using only their audio streams.
Methods
The proposed system employs a two-level architecture:
- Feature Extraction: They extract four groups of features:
- Acoustic Space: This standard approach uses Mel-Frequency Cepstral Coefficients (MFCCs) and Perceptual Linear Predictive (PLP) coefficients, modeled by Gaussian Mixture Models (GMMs). They also explore Factor Analysis (FA) to reduce intra-genre variability, showing significant improvements.
- Speaker Interactivity: This novel feature group captures the dynamics of speech, including speaker turn density, number of speakers, and main speaker's speaking time, extracted using a speaker diarization system.
- Speech Quality: Features derived from speech recognition confidence scores, linguistic probabilities, and phonetic entropy are used to assess speech clarity and predictability.
- Acoustic Instability: This group includes features like Short-Term Energy (STE), Zero Crossing Rates (ZCR), and Acoustic Model Breaking (AMB) to capture temporal variations in the audio signal.
- Model Combination:
- Level 1: Each feature group is initially classified using genre-specific GMMs.
- Level 2: The probability outputs from the Level 1 classifiers are combined into a vector, which is then fed into a Support Vector Machine (SVM) for the final genre classification. They also compare this "model-level combination" with a "feature-level combination" where all raw features are directly fed into an SVM.
Datasets
The authors used a corpus of 1200 videos from Dailymotion and YouTube, with durations between 2 to 5 minutes. The corpus is balanced across five genres: news, movies, cartoons, music, and commercials. 1000 videos were used for training and 200 for testing. The speech content is in French.
Key Insights and Ideas
- Complementarity of Features: While acoustic space features (especially PLP with FA) are highly discriminative, higher-level features like speaker interactivity and speech quality offer complementary information that significantly boosts overall performance.
- Factor Analysis for Robustness: Applying Factor Analysis to acoustic space features effectively reduces intra-genre variability, leading to substantial improvements in identification rates.
- Interactivity's Niche: Speaker interactivity features, though less discriminative on their own, prove crucial for improving the classification of genres like commercials and cartoons, which were poorly recognized by acoustic space methods alone.
- Model vs. Feature Combination: Both model-level (combining GMM probabilities) and feature-level (combining raw features) SVM approaches yield similar average performances. However, model-level combination seems to benefit more from interactivity features, while feature-level combination shows better gains from speech quality features. This suggests that genre-specific combination strategies might be beneficial.
Key Results
- The best performing system, combining all four feature groups using model-level combination with an SVM, achieved an overall identification rate of 91%.
- Acoustic space characterization using PLP coefficients with Factor Analysis alone reached 86% accuracy.
- Speaker interactivity features, while low on their own (64% accuracy), significantly improved the performance on commercials and cartoons when combined with acoustic space features, leading to a balanced recognition rate above 84% for all genres.
- The final system's performance is comparable to audio-video combination methods, highlighting the power of audio-only analysis.
Limitations and Future Work
- Speech Quality Complementarity: The speech quality features showed less complementarity than expected, and in some combinations, even led to a decrease in performance. The authors attribute this to the SVM's potential difficulty in effectively handling posterior probabilities from GMMs.
- Genre Specificity: The paper suggests that genre-specific combination strategies could further improve classification accuracy.
- Future Directions: The authors plan to generalize their method to a larger set of genres and explore various combination strategies.
Summary updated