Comparing and Combining Generative and Posterior Probability Models: Some Advances in Sentence Boundary Detection in Speech

Yang Liu, Andreas Stolcke, Elizabeth Shriberg, Mary P. HarperEMNLP 2004

Identifying sentence boundaries in spoken language is a fundamental challenge in speech processing. Unlike written text, which utilizes punctuation and capitalization, raw speech recognition output often resembles a continuous "word salad." This paper explores advanced modeling techniques to segment these streams into Sentence-like Units (SUs), a critical step for downstream tasks such as machine translation and summarization.

The Challenge of Sentence Boundary Detection

Detecting boundaries requires the integration of two distinct types of information: lexical/syntactic features (the words themselves) and prosodic features (the acoustic cues like pauses, pitch changes, and energy). Traditional Hidden Markov Models (HMMs) are effective for sequence modeling but struggle to incorporate high-dimensional, overlapping features without significant mathematical complexity.

System architecture for sentence boundary detection, showing the fusion of textual and prosodic features.

As shown in the system architecture above, the process involves extracting features from both the raw acoustic signal and the word strings generated by a speech recognizer. These knowledge sources are then fused to generate a sentence boundary hypothesis.

Modeling Approaches

The researchers compared two primary modeling paradigms:

  1. Generative Model (HMM): This baseline model treats sentence boundaries as hidden states that produce observable data. It utilizes N-gram language models for transitions and decision-tree-processed prosodic features as observation likelihoods.
  2. Discriminative Model (Maxent): The Maximum Entropy model directly estimates the probability of a boundary. Its primary advantage is the ability to handle hundreds of overlapping, correlated features (like POS tags and phrase chunks) simultaneously.

To optimize the Maxent model, the authors introduced a cumulative binary coding scheme. Instead of using raw probability scores, they converted them into a series of binary "thermometer" flags (e.g., is_prob > 0.1, is_prob > 0.3). This allowed the model to capture non-linear relationships more effectively.

Finally, a Combined Model was developed by averaging the posterior probabilities of the HMM and Maxent systems, aiming to leverage the global sequence modeling of the HMM and the local feature depth of the Maxent model.

Experimental Results

The models were evaluated on Broadcast News (BN) and Conversational Telephone Speech (CTS) datasets, using both human-verified reference transcripts (REF) and noisy Speech-to-Text (STT) outputs.

Comparison of error rates (%) for HMM, Maxent, and Combined models across different datasets and transcript types.

As illustrated in the results table, the Combined Model consistently outperformed the individual systems across all categories. On CTS reference transcripts, the combined approach reduced the error rate to 29.30%, representing a significant relative error reduction over the HMM baseline.

Key findings include:

  • Complementary Strengths: Maxent excelled in text-only classification due to its feature-rich nature, while the HMM remained competitive by integrating prosody throughout the entire sequence.
  • Robustness of Prosody: Prosodic features, particularly pause duration, proved more resilient to speech recognition errors than lexical features.
  • The Noise Penalty: Error rates increased by approximately 10-12% when moving from perfect transcripts to noisy STT output, highlighting the dependency of boundary detection on the quality of the underlying word recognition.

Conclusion

The study demonstrates that combining generative and discriminative models provides a robust framework for speech segmentation. While the Maxent model's local window and binary thresholding have limitations, the hybrid approach successfully mitigates these weaknesses. The authors' suggestion of Conditional Random Fields (CRFs) as a future direction proved prescient, as CRFs eventually became a standard for combining discriminative power with sequence modeling in natural language processing.

Summary updated

BibTeX
@inproceedings{liu2004comparing,
  author = {Yang Liu and Andreas Stolcke and Elizabeth Shriberg and Mary P. Harper},
  title = {{Comparing and Combining Generative and Posterior Probability Models: Some Advances in Sentence Boundary Detection in Speech}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2004},
}
Figure