Accounting for prosodic information to improve ASR-based topic tracking for TV broadcast news

Camille Guinaudeau, Julia HirschbergInterspeech 2011

TL;DR

This paper demonstrates that incorporating prosodic features (like pitch and energy) into Automatic Speech Recognition (ASR) systems significantly improves their ability to track topics in TV broadcast news. By analyzing how speakers emphasize words, the system can better identify semantically related segments, leading to more effective content navigation.

What Problem Does This Paper Address?

The explosion of online video content, particularly TV news, presents a challenge for users trying to find specific information. Topic tracking aims to automatically identify segments of a video that discuss the same subject. Traditional methods rely heavily on the textual content of transcripts, often using vector space models to measure word overlap. However, TV news is inherently multimodal, and the way speech is delivered – its prosody – carries crucial information about emphasis, newness, and importance. This paper addresses the gap of not fully leveraging these prosodic cues in ASR-based topic tracking systems, particularly for French broadcast news.

Methods

The authors propose integrating prosodic information into an existing ASR-based topic tracking system. The core idea is to use prosody to re-weight words in the transcript, giving more importance to emphasized or prominent words, even if they aren't statistically frequent in a standard TF-IDF (Term Frequency-Inverse Document Frequency) analysis.

Two main approaches for extracting prosodic information are explored:

  • Acoustic Information Extraction (AIE): This method directly extracts acoustic features like energy (RMS) and fundamental frequency (f0) from the audio signal. These features are then z-score normalized per speaker and aligned with the ASR transcript. Different strategies are used to aggregate prosodic values for a given lemma (word stem), including taking the maximum (MAX) or average (AVE) acoustic value across its occurrences.
  • Prosodic Event Detection (PED): This technique uses a trained system (AuToBI) to identify specific prosodic events like pitch accents and boundary tones. These events are then associated with words, providing a probability score for a word being prominent.

These prosodic scores are then combined with traditional TF-IDF scores. The combination can be done by simply averaging or by using a weighted sum, where parameters (θir and θac) control the relative importance of TF-IDF and prosodic information.

Datasets

  • Reference Corpus for TF-IDF: 800,000 articles from the French newspaper Le Monde (1987-2003). This corpus is used to compute IDF scores and train the ASR system's language model.
  • Training Corpus for PED: The C-PROM Corpus, a manually annotated corpus for French prosodic prominence studies.
  • Test Corpus: 8 TV news programs broadcasted in February and March 2007 on the French television channel France 2, totaling approximately 4 hours of content. This corpus was manually segmented into 177 topic segments.

Key Insights and Ideas

  • Prosody Matters for Topic Tracking: The most significant insight is that prosodic information, when properly integrated, can substantially boost the performance of ASR-based topic tracking systems, even outperforming TF-IDF alone in some configurations.
  • Emphasis Signals Importance: Words that are spoken with higher energy or pitch, even if not statistically common, often carry important thematic information. Prosody helps to "boost" these words in the topic tracking model.
  • AIE vs. PED Performance: The direct acoustic feature extraction (AIE) generally performed better than the prosodic event detection (PED) method. This is hypothesized to be due to the PED system being trained on a different corpus (C-PROM) than the test data, whereas AIE is directly extracted from the test data without a specific learning phase for prosody.
  • MAX Strategy is Key: For aggregating prosodic values of a lemma, the MAX strategy (taking the highest acoustic value) yielded significantly better results than averaging or other methods. This suggests that a single strong prosodic cue is more indicative of importance than an average.
  • Combined Approach is Superior: The joint use of TF-IDF and prosodic information consistently outperformed using either cue in isolation, highlighting the complementary nature of these signals.

Key Results

  • The integration of prosodic information using the AIE method improved the F1-measure by 13 points compared to the baseline TF-IDF system.
  • Using the PED technique improved the F1-measure by 8 points.
  • The best combined result was achieved when the TF-IDF and AIE scores were weighted equally (θir = θac = 1-1), yielding an F1-measure of 48.40%. This is a substantial improvement over the baseline TF-IDF system's F1 of 35.24%.
  • The MAX strategy for AIE, combined with the intensity feature, achieved an F1-measure of 38.80% when used alone.

Limitations and Future Work

  • PED Training Data: The authors hypothesize that the lower performance of the PED technique is related to the mismatch between its training corpus (C-PROM) and the news corpus.
  • ASR Transcription Errors: The paper acknowledges that transcription errors and the extensive use of synonyms can degrade topic tracking quality.
  • Future Work:
    • Train the PED system on a news corpus to see if performance improves.
  • Expand the study to a larger and more diverse corpus of TV programs (e.g., current affairs reports). Preliminary results on a larger corpus show continued improvements.
  • Integrate other cues, such as ASR confidence measures (to mitigate transcription errors) and semantic information (to handle synonyms).

Summary updated

BibTeX
@inproceedings{guinaudeau2011accounting,
  author = {Camille Guinaudeau and Julia Hirschberg},
  title = {{Accounting for prosodic information to improve ASR-based topic tracking for TV broadcast news}},
  booktitle = {Interspeech},
  year = {2011},
}
Figure