Integration of context-dependent durational knowledge into HMM-based speech recognition

Xue Wang, Louis F. M. ten Bosch, L.c.w. Pols4th 1996 · DOI

TL;DR

This paper tackles the underutilization of phonetic duration information in standard HMM-based speech recognition systems. By deriving theoretical relationships between HMM parameters and duration distributions, and then integrating context-dependent duration models into a post-processing rescoring step, the authors demonstrate a pathway to improve recognition accuracy.

What Problem Does This Paper Address?

Traditional Hidden Markov Models (HMMs) used in Automatic Speech Recognition (ASR) often struggle to effectively incorporate phonetic duration information. While duration is a crucial acoustic cue for speech perception, standard HMMs, especially those with simple state structures, don't inherently model it well. Techniques like Hidden Semi-Markov Models (HSMMs) exist but are computationally complex. This paper aims to bridge this gap by integrating durational knowledge into standard HMM frameworks, specifically focusing on context-dependent duration statistics, without resorting to computationally prohibitive models.

Methods

The authors propose a two-pronged approach:

  1. Theoretical Analysis of HMM Durational Behavior: They derive closed-form expressions for the duration probability distribution (DPDF) of whole HMMs, particularly for left-to-right topologies. This analysis reveals that even simple linear HMMs possess rich durational modeling capabilities, and they establish theoretical links between HMM parameters (like self-loop probabilities) and the resulting DPDF.
  2. Integration of Context-Dependent Duration Models:
    • Constrained ML Training: They explore modifying the standard Baum-Welch Maximum Likelihood (ML) training to incorporate durational statistics (mean and variance) as constraints. This involves solving a system of non-linear equations using Newton-Raphson procedures.
  3. Post-processing Rescoring: The primary integration method involves using context-dependent duration models (derived from statistical analysis of TIMIT) to rescore N-best transcriptions. This is a two-phase process: first, generating N-best word hypotheses, and second, generating phone-level transcriptions and then applying an external duration model to score them. They also introduce a word-juncture model to handle phone realization variations at word boundaries.

Datasets

  • TIMIT: This multi-speaker database is used extensively for both training and statistical analysis of context-dependent durational statistics. Both the training (sx, si) and test sets are utilized.

Key Insights and Ideas

  • HMMs are richer than they seem: Even basic linear HMMs, when analyzed correctly, offer significant potential for modeling phonetic segment durations, going beyond just minimum duration constraints.
  • Duration is context-dependent: The paper highlights the importance of context (speaking rate, stress, syllable position) in shaping phone durations, necessitating context-dependent modeling.
  • Post-processing offers a flexible integration path: Integrating durational knowledge as a post-processing step (rescoring N-best lists) provides a relatively simple way to leverage this information without fundamentally altering the core HMM recognition engine. This approach can potentially correct errors made by earlier stages.
  • Word junctures are critical: Phone realizations at word boundaries are a significant source of variation, and modeling these junctures is crucial for accurate phone-level duration analysis.

Key Results

  • Constrained Training: The paper reports improvements in word-correct scores and segmentation accuracy when using HMMs trained with durational constraints compared to standard ML training. For example, word-correct scores improved from 80.61% to 86.83% (and segmentation accuracy from 83.48% to 84.48%) on a subset of the TIMIT data.
  • Post-processing Rescoring: While the integration of context-dependent duration models via N-best rescoring showed a "tiny increment" in word-correct score (3 more words correct) on a specific "wrong set" of utterances, the authors acknowledge limitations in their N-best algorithm. They anticipate that experiments on the whole dataset might show a decrease in score due to inaccuracies in the phone-level transcription generation.

Limitations and Future Work

  • N-best Algorithm Limitations: The primary limitation acknowledged is the lack of a good N-best algorithm, which hindered the effectiveness of the post-processing rescoring approach. The separation of word-level and phone-level transcriptions also introduced inaccuracies.
  • Computational Complexity: While avoiding HSMMs, the constrained ML training still involves solving non-linear equations, which can be computationally intensive.
  • Future Work: The authors suggest implementing their context-dependent duration models with a better N-best algorithm and systematically optimizing various system components. They also emphasize the potential of integrating other long-term features like pitch into this framework.

Summary updated

Abstract

HMMThis paper presents research on integrating context-dependent durational knowledge into HMM-based speech recognition.The first part of the paper presents work on obtaining relations between the parameters of the context-free HMMs and their durational behaviour, in preparation for the context-dependent durational modelling presented in the second part.Duration integration is realised via rescoring in the post-processing step of our N-best monophone recogniser.We use the multi-speaker TIMIT database for our analyses.The single-state dpdf (which is geometrical) is less important than the dpdf of the whole HMM, because in actual practice it is the latter that models a phonetic segment.In this section we firstly derive the closed-form whole-model dpdf for general leftto-right HMM.Left-to-right is by far the most common type of transition topology used for speech recognition.It may include any number of skipping transitions and parallel paths but no feedback loops that contains more than one state.Then an analysis will be given of the general properties of the dpdf, with the help of some examples of useful topologies.
BibTeX
@inproceedings{wang1996integration,
  author = {Xue Wang and Louis F. M. ten Bosch and L.C.W. Pols},
  title = {{Integration of context-dependent durational knowledge into HMM-based speech recognition}},
  booktitle = {4th},
  year = {1996},
  doi = {10.21437/icslp.1996-282},
}
Figure