Prominence based scoring of speech segments for automatic speech-to-speech summarization

Sree Harsha Yella, Vasudeva Varma, Kishore PrahalladInterspeech 2010

TL;DR

This paper introduces a novel method for automatic speech summarization that relies on "prominence" – the perceptual salience of syllables – rather than traditional ASR transcripts or human-labeled summaries. By analyzing syllable prominence, the system can effectively rank speech segments for inclusion in a summary, achieving comparable or even better results than existing methods without requiring extensive linguistic annotations.

What Problem Does This Paper Address?

Creating automatic summaries of spoken content (speech summarization) is challenging. Traditional approaches often rely on either:

  1. ASR Transcripts: These systems use the output of Automatic Speech Recognition (ASR) to identify important segments based on text-based features like TF-IDF. However, ASR systems can be error-prone, especially for less common languages or noisy audio, and their accuracy directly impacts summary quality.
  2. Human-Labeled Summaries: Other methods train supervised models using "gold standard" human-created summaries. While effective, creating these summaries is time-consuming and resource-intensive, making it impractical for many languages or new domains.

The core problem this paper addresses is the dependency on ASR transcripts or human-labeled data for effective speech summarization. The authors aim to develop a method that can generate good summaries in an unsupervised manner, leveraging inherent prosodic features of speech itself.

Methods

The paper proposes a prominence-based scoring mechanism for ranking speech segments. Here's a breakdown of their approach:

  • Prominence Measurement: The core idea is that prominent syllables (those that "stand out") carry important information. They use a method described in prior work [10] to compute a "prominence value" for each syllable. This value is derived from acoustic features like syllable duration, subband energy, and pitch variation.
  • Segment Scoring: Once syllable prominence values are calculated for a speech segment, the segment itself is assigned a score. The paper explores four different scoring functions:
  • Mean Prominence (mp): The average prominence value of all syllables in the segment.
  • Maximum Prominence (Mp): The highest prominence value among syllables in the segment.
  • Mean Difference (mdp): The average absolute difference in prominence between consecutive syllables. This aims to capture prosodic variations.
  • Maximum Difference (Mdp): The maximum absolute difference in prominence between consecutive syllables.
  • Summarization Process: Speech segments are ranked based on their scores (using the best-performing function, mdp). The top-ranked segments are then concatenated chronologically to form the summary.
  • Unsupervised Nature: Crucially, this method does not require ASR transcripts or human-labeled summaries for training. It directly uses acoustic features and syllable prominence.

Datasets

The research was conducted on two distinct speech corpora:

  1. Boston University Radio News Corpus (BURNC): A subset containing 40 read-style news stories spoken by a female speaker. This corpus includes orthographic text transcripts and, importantly for their initial experiments, hand-labeled pitch accent markings (treated as prominence markers).
  2. Switchboard Data Corpus (ICSI release): A subset of 40 spontaneous telephone conversations about credit cards. This corpus includes orthographic text transcripts and speaker turn information.

Key Insights and Ideas

  • Prominence as a Proxy for Importance: The paper demonstrates that prominent syllables are more likely to be content words, and segments with higher average prominence scores tend to be more important for summarization. This suggests that prosody, specifically prominence, carries significant semantic weight.
  • Unsupervised Summarization is Feasible: The most significant insight is that high-quality speech summaries can be generated without relying on ASR or human annotations, which are major bottlenecks in current systems.
  • Prominence Captures Complementary Information: When combined with TF-IDF scores (derived from transcripts), prominence-based ranking yields even better summaries, indicating that prosodic features capture information not present in lexical content alone.
  • Automatic Syllable Boundary Detection is Sufficient: For the unsupervised approach, they developed an automatic method for identifying syllable boundaries, which is crucial for calculating segment prominence without manual annotation.

Key Results

  • Performance Comparable to Supervised Methods: The prominence-based scoring method, particularly using the mdp function, achieved ROUGE scores comparable to systems trained on ASR transcripts (TF-IDF) and supervised systems trained on gold-standard summaries. For example, on the BURNC corpus, the mdp method achieved ROUGE-1 F-measures of 0.508, very close to the supervised system's 0.478 and TF-IDF's 0.514.
  • Combined Approach Outperforms Individuals: Combining prominence scores with TF-IDF scores resulted in improved ROUGE scores compared to using either method alone, highlighting the complementary nature of prosodic and lexical information.
  • Robustness to Compression Ratio: The system demonstrated good performance across various compression ratios, with precision dropping only slightly as the desired summary length decreased.
  • Effectiveness on Spontaneous Speech: The method also showed promising results on the Switchboard corpus (spontaneous dialogues), achieving ROUGE scores comparable to TF-IDF and supervised methods, even when using speaker turns as the basic summarization unit.

Limitations and Future Work

  • Reliance on Human Prominence Markings (Initial Experiments): While the core contribution is unsupervised, the initial validation of prominence's usefulness relied on hand-labeled prominence markings from human experts. The paper then moves to an unsupervised approach using automatic syllable boundary detection.
  • Automatic Syllable Boundary Errors: The automatic syllable boundary detection method had a missed detection rate (MDR) of 12.3% and a false alarm rate (FAR) of 9.4%. While the paper shows good results despite these errors, further improvements in syllable boundary detection could enhance performance.
  • Future Work: The authors suggest exploring the combination of prominence-based scores with TF-IDF scores more deeply. They also mention that the output summaries are presented in speech form, preserving the characteristics of the input speech signal, which is a key aspect of speech-to-speech summarization.

Summary updated

BibTeX
@inproceedings{yella2010prominence,
  author = {Sree Harsha Yella and Vasudeva Varma and Kishore Prahallad},
  title = {{Prominence based scoring of speech segments for automatic speech-to-speech summarization}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure