How prosody improves word recognition

Ken Chen, Mark Hasegawa‐Johnson2004 · DOI

TL;DR

This paper argues that prosody, often ignored in speech recognition, can significantly improve word recognition accuracy. They propose a Bayesian network model and an information-theoretic analysis to show how jointly modeling prosody with words enhances the speech recognition system, achieving a 2.5% accuracy improvement on a news corpus.

What Problem Does This Paper Address?

Traditional Automatic Speech Recognition (ASR) systems have largely treated prosody—the rhythm, stress, and intonation of speech—as irrelevant for word recognition. While prosodic features are known to be influenced by higher-level linguistic elements like syntax and meaning, and are crucial for human comprehension, their direct application to improving word-level accuracy in ASR has been limited. Existing attempts to incorporate prosody, such as using normalized energy or duration, have yielded only marginal improvements. This paper aims to bridge this gap by demonstrating how prosody can be effectively integrated into ASR frameworks to boost word recognition performance.

Methods

The authors propose a Bayesian network to model the complex interdependencies between acoustic-phonetic features (X), acoustic-prosodic features (Y), word sequences (W), prosody sequences (P), syntax sequences (S), and meaning (M). They then introduce a prosody-dependent speech recognition framework. This framework models the joint probability of words and prosody, treating them as a single unit.

The core of their approach lies in an information-theoretic analysis that proves the mutual information between acoustic observations and correct word hypotheses can be increased by jointly modeling prosody. They define a model discriminant function and show that a prosody-dependent recognizer improves performance if a specific condition related to the ratio of prosody probabilities is met. This condition essentially means that the system performs better if the correct prosodic sequence is well-predicted by the word transcription, and the acoustic observations are well-predicted by the prosody.

Datasets

The experiments were conducted on the Boston University Radio News Corpus. This corpus is notable for its rich annotations, including orthographic transcriptions, phone alignments, part-of-speech tags, and prosodic labels (using the ToBI system). The authors simplified the ToBI labels to distinguish between accented/unaccented and phrase-medial/phrase-final prosodic variations for each word.

Key Insights and Ideas

  • Prosody is not just "noise" for word recognition: Contrary to common belief, prosody carries information that can disambiguate word sequences and improve recognition accuracy.
  • Joint modeling is key: The effectiveness of prosody comes from modeling it jointly with words, not as an independent feature. This allows the system to leverage the strong dependencies between word sequences and their prosodic realization.
  • Bayesian network provides a theoretical foundation: The Bayesian network elegantly illustrates how prosody interacts with other linguistic levels, providing a principled way to design ASR systems.
  • Information-theoretic justification: The paper provides a rigorous mathematical proof that incorporating prosody can increase the information conveyed by acoustic observations about the correct word sequence.
  • Practical implications for ASR design: The insights suggest specific ways to maximize the benefits of prosody, such as focusing on prosodic labels predictable from word sequences and acoustic features well-predicted by prosody.

Key Results

The proposed prosody-dependent speech recognition system achieved a 2.5% improvement in word recognition accuracy (WRA) on the Boston University Radio News Corpus compared to a standard prosody-independent system. Specifically, the most advanced recognizer (RDC), which combined a prosody-dependent acoustic model with a prosody-dependent factorial language model, showed significant gains. Beyond word recognition, the system also demonstrated substantial improvements in recognizing pitch accents and intonational phrase boundaries.

Limitations and Future Work

The authors acknowledge that their current system does not explicitly model syntax due to the small corpus size. They suggest that future work could explore incorporating syntax into the Bayesian network and the ASR framework. Additionally, the prosodic labeling in their experiments was simplified; a more detailed ToBI labeling could potentially yield further improvements. The paper also hints at future work in optimizing the selection of prosodic features and acoustic cues that are most sensitive to prosodic variations.

Summary updated

Abstract

Prosody has been traditionally regarded as useless for word recognition.In this paper, we provide a schematic view describing how prosody can help word recognition.We provide our view in terms of a Bayesian network that models the stochastic dependence among acoustic observation, word, prosody, syntax and meaning, and an information-theoretic analysis proving that the mutual information between acoustic observation and correct word hypotheses improves if prosody is jointly modeled with word in a prosody dependent speech recognition framework.We also report our experiment on Radio News Corpus in which prosody has improved word recognition accuracy by 2.5%.
BibTeX
@misc{chen2004how,
  author = {Ken Chen and Mark Hasegawa‐Johnson},
  title = {{How prosody improves word recognition}},
  year = {2004},
  doi = {10.21437/speechprosody.2004-134},
}
Figure