Acoustic event detection for spotting "hot spots" in podcasts

Kouhei Sumi, Tatsuya Kawahara, Jun Ogata, Masataka GotoInterspeech 2009

TL;DR

This paper tackles the challenge of automatically identifying "hot spots" in podcasts – segments that listeners would find particularly interesting. They achieve this by developing a novel acoustic event detection system that adaptively adjusts its segmentation based on the audio environment (speech, music, or a mix), leading to improved detection of key audible reactions like laughter and reactive tokens.

What Problem Does This Paper Address?

The explosion of audio content like podcasts presents a significant challenge for users: how to efficiently find interesting segments without listening to hours of material. Traditional approaches rely on Automatic Speech Recognition (ASR) to transcribe audio, making it searchable. However, ASR struggles with the diverse recording environments, background music, spontaneous speech, and conversational nature of podcasts. This paper addresses this gap by proposing a method to detect "hot spots" not by transcribing speech, but by identifying meaningful non-verbal audible reactions (like laughter and specific vocalizations) that signal listener engagement. These reactions often precede segments that listeners would find interesting.

Methods

The core of the proposed method lies in an improved acoustic event detection (AED) system that leverages Bayesian Information Criterion (BIC) based segmentation. The novelty lies in adaptively switching the penalty weight (λ) for the BIC segmentation based on the audio's characteristics.

Here's a breakdown:

  • Initial Classification: The system first uses Gaussian Mixture Models (GMMs) to classify the audio stream into three broad categories: speech, music, and a mix of speech and music.
  • Adaptive BIC Segmentation: For each classified segment, a specific penalty weight (λ) is applied to the BIC-based segmentation algorithm. This is crucial because music segments tend to be more varied and might be over-segmented with a standard penalty, while mixed segments are harder to segment due to smearing effects. The paper estimates optimal λ values for speech (λ_spe), music (λ_mus), and mix (λ_mix) based on the dispersion of Gaussian distributions within trained GMMs.
  • Acoustic Event Detection: After segmentation, GMMs are used to classify specific acoustic events within these segments, including laughter, speech (male/female), music, silence, and reactive tokens.
  • Reactive Token Detection Enhancement: For reactive tokens (like "hu:N", "he:", "a:") which indicate listener interest, the system combines filled pause detection (identifying prolonged vowels) with a simple ASR to distinguish these tokens from other fillers.

Datasets

  • Training Data: The system was trained using a combination of existing datasets:
  • JNAS (Japanese Speech Corpus)
    • RWC-MDB (Real World Computing Music Database)
    • IMADE corpus
    • Web data
  • Evaluation Data: The system was evaluated on eight podcast episodes from four different programs. Two sets of podcasts were used for evaluation:
  • Program-open set: 19 episodes from programs not used in training.
  • Program-closed set: 23 episodes, including one previous episode from each program used in the test set. This set is designed to assess performance in scenarios with similar speakers and music.

Key Insights and Ideas

  • Context-Aware Segmentation is Key: The most significant insight is that a "one-size-fits-all" approach to audio segmentation is insufficient for complex audio like podcasts. Adapting segmentation parameters (like the BIC penalty weight) based on the audio's nature (speech, music, mix) dramatically improves accuracy.
  • Non-Verbal Cues as "Hot Spot" Indicators: The paper demonstrates that focusing on audible reactions (laughter, reactive tokens) is a viable and effective strategy for identifying "hot spots" without relying solely on ASR. This opens up new avenues for content analysis in audio media.
  • Trade-offs in Reactive Token Detection: The detection of reactive tokens involves a trade-off between recall and precision. The authors acknowledge that their strict constraints for rejecting fillers lead to lower recall but higher precision, which they argue is acceptable for identifying distinct "hot spots."

Key Results

  • Improved Frame-wise Accuracy: The proposed adaptive method achieved significantly better frame-wise accuracy (segmentation and classification of eight classes) compared to using a constant penalty weight. For the program-open set, accuracy reached 74.5%.
  • Laughter Detection: The proposed method improved laughter detection, achieving an F-measure of 68.7%. While recall was slightly degraded for subtle laughter, it effectively detected loud laughter, which is more indicative of listener engagement.
  • Reactive Token Detection: The system achieved an F-measure of 64.0% for reactive token detection, outperforming constant penalty weight approaches.
  • Program-Closed Set Advantage: The system showed a drastic improvement in accuracy on the program-closed set, highlighting the benefit of using prior episodes from the same program due to consistent speakers and music.

Limitations and Future Work

  • Limitations:
    • The recall for laughter detection was degraded for subtle laughter, which might be a concern for some applications.
  • The strict constraints for rejecting fillers and falters in reactive token detection led to lower recall.
  • Future Work:
    • Incorporating clustering of speech segments for speaker diarization (identifying different speakers).
  • Designing and implementing a "hot spot browser" to leverage the developed detection capabilities.

Summary updated

BibTeX
@inproceedings{sumi2009acoustic,
  author = {Kouhei Sumi and Tatsuya Kawahara and Jun Ogata and Masataka Goto},
  title = {{Acoustic event detection for spotting "hot spots" in podcasts}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure