Integrating Prosodic and Lexical Cues for Automatic Topic Segmentation

Gökhan Tür, Dilek Hakkani-Tür, Andreas Stolcke, Elizabeth Shriberg

TL;DR

Researchers from Bilkent University and SRI International demonstrate that combining lexical cues (the words used) with prosodic cues (speech patterns like pitch and pauses) significantly improves automatic topic segmentation in broadcast news. By integrating these two streams into a unified probabilistic model, they achieved up to a 27% reduction in error compared to traditional word-only methods.

What Problem Does This Paper Address?

Imagine reading a book with no punctuation, no paragraph breaks, and no capitalized letters. That is essentially what a raw transcript from an Automatic Speech Recognizer (ASR) looks like. For a machine to make sense of a continuous stream of broadcast news, it first needs to perform topic segmentation: dividing the audio into topically coherent stories.

Traditional systems rely almost exclusively on "lexical" cues—looking for shifts in vocabulary. However, this approach has two major flaws:

  1. ASR Errors: If the speech recognizer mishears words, the segmenter gets confused.
  2. Missing Cues: Speech doesn't have headers or "The End" markers.

The authors argue that speech contains a hidden layer of information: prosody. Humans use changes in pitch, duration, and pauses to signal the end of a thought or a change in subject. This paper explores how to teach machines to use these non-lexical signals to find topic boundaries more accurately.

Methods

The authors propose a two-phase approach:

  1. Chopping: The continuous audio is first cut into "pseudo-sentences." They tested several criteria (fixed word counts, speaker turns, pauses) and found that pauses longer than 0.575 seconds were the most effective "chopping" points.
  2. Segmentation: They then classify each chop-point as either a "topic boundary" or "not a boundary" using two models:
  3. Lexical Model: A Hidden Markov Model (HMM) with 100 topic clusters. It uses unigram language models to detect when the "vocabulary" of the stream shifts significantly.
  4. Prosodic Model: A CART-style decision tree that looks at features like pause duration, F0 (pitch) resets, speaker gender, and turn duration.

The "Secret Sauce" (Model Combination): The core innovation is how they fused these models. They tested two methods:

  • Feature-level fusion: Feeding the HMM's probability as an extra feature into the decision tree.
  • Model-level fusion: Using the decision tree to provide "likelihoods" for boundary states within the HMM. This second approach proved superior because it allowed for better global optimization using the Viterbi algorithm.

Datasets

The study utilized the 1997 LDC Broadcast News (BN) corpus (approximately 70 hours for training and 3 hours for testing). Evaluation was conducted using the DARPA-TDT (Topic Detection and Tracking) metrics, which account for both "misses" (missing a boundary) and "false alarms" (inventing a boundary).

Key Insights and Ideas

  • Prosody is Robust: Unlike word-based models, prosodic features are relatively unaffected by ASR errors. Even if the system doesn't know what was said, it can tell how it was said.
  • The Power of the Pause: Pause duration was the single most important prosodic feature. However, pitch (F0) "resets"—where a speaker starts a new topic at a higher pitch—were crucial for distinguishing between a simple sentence break and a true topic shift.
  • Complementary Strengths: Lexical models tend to have more "misses," while prosodic models have more "false alarms" (because regular sentences often sound like mini-topic ends). Combining them allows the lexical model to "filter" the prosodic model's over-eagerness.
  • Gender and Style Matter: Interestingly, the decision tree used speaker gender as a feature. The authors found that male speakers in the corpus were more prone to certain types of pausing, suggesting that prosodic models may need to be "speaker-aware" to be truly effective.

Key Results

  • Prosody Alone is Competitive: A model using only prosodic features performed nearly as well as (and sometimes better than) a standard word-based HMM.
  • Significant Gains from Fusion: The combined HMM-based model achieved a 24% to 27% relative improvement in segmentation cost over the word-only baseline.
  • Resilience to Noise: When moving from perfect "true" transcripts to messy ASR transcripts (with a 30.5% word error rate), the combined model's performance remained remarkably stable, proving its real-world utility.

Limitations and Future Work

  • Source Specificity: The models were tuned to specific news programs (e.g., CNN vs. ABC). The authors note that different shows have different "prosodic signatures," and more work is needed to make these models generalize across different broadcast styles.
  • Suboptimal Training: The HMM was trained in a somewhat supervised fashion. The authors suggest that using unsupervised methods like the Baum-Welch algorithm could further refine the topic clusters.
  • Beyond Pauses: While pauses and pitch were the focus, other features like speaking rate and energy (volume) patterns were not fully exploited and could offer further improvements.

Summary updated

Abstract

We present a probabilistic model that uses both prosodic and lexical cues for the automatic segmentation of speech into topically coherent units. We propose two methods for combining lexical and prosodic information using hidden Markov models and decision trees. Lexical information is obtained from a speech recognizer, and prosodic features are extracted automatically from speech waveforms. We evaluate our approach on the Broadcast News corpus, using the DARPA-TDT evaluation metrics. Results show that the prosodic model alone is competitive with word-based segmentation methods. Furthermore, we achieve a significant reduction in error by combining the prosodic and word-based knowledge sources.
BibTeX
@inproceedings{tur2001integrating,
  author = {Gökhan Tür and Dilek Hakkani-Tür and Andreas Stolcke and Elizabeth Shriberg},
  title = {{Integrating Prosodic and Lexical Cues for Automatic Topic Segmentation}},
  booktitle = {Comput. Linguistics},
  year = {2001},
  url = {https://openreview.net/forum?id=FVocljQySS7},
}

Loading related content...

Figure