About handling boundary uncertainty in a speaking rate dependent modeling approach

Denis Jouvet, Dominique Fohr, Irina IllinaInterspeech 2011

TL;DR

This paper tackles the challenge of accurately modeling speech variations, specifically speaking rate, by addressing uncertainty in speech segment boundaries. They propose methods to incorporate this boundary uncertainty into both the training and decoding phases of speech recognition systems, leading to improved accuracy.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are notoriously sensitive to variations in speech, such as speaker differences, background noise, and speaking style. One significant factor is speaking rate – how fast someone speaks. Previous work has shown that modeling speaking rate can improve ASR performance. However, accurately estimating speaking rate relies on knowing the precise boundaries of speech segments (like words or phones). These boundaries are often uncertain due to the nature of ASR systems themselves, leading to errors in speaking rate estimation and, consequently, suboptimal ASR performance. This paper aims to bridge this gap by explicitly handling the uncertainty in speech segment boundaries when modeling speaking rate.

Methods

The core of this work lies in adapting a speaking rate dependent modeling framework to account for boundary uncertainty. The authors build upon a variability-dependent modeling approach where the probability of a pronunciation variant of a word is conditioned on the speaking rate.

Here's a breakdown of their approach:

  • Variability-Dependent Modeling: The standard ASR decoding process finds the most likely word sequence given acoustic observations. This paper extends this by incorporating a variability variable (like speaking rate) into the model. The probability of a phone sequence (representing a pronunciation) is then conditioned on both the word sequence and the speaking rate.
  • Handling Boundary Uncertainty in Training: Instead of assigning a single speaking rate to a speech segment, the authors propose a method to distribute the "counts" of pronunciation variant occurrences across adjacent speaking rate bins. This is done by considering a range of possible speaking rates derived from the estimated segment duration plus/minus a boundary uncertainty value ($\delta$). This effectively smooths the speaking rate distributions and provides more data for less frequent extreme speaking rates.
  • Handling Boundary Uncertainty in Decoding: In the decoding phase, instead of using a single estimated speaking rate, the system now considers a weighted sum of probabilities for different speaking rate bins (including the estimated rate and the rates derived from the uncertainty). This probabilistic approach allows the decoder to explore multiple possibilities for the speaking rate.

Datasets

  • Training Data: French ESTER2 speech corpus (approximately 180 hours of signal and 2 million running words). Counts of pronunciation variants were estimated using correctly recognized words from this training set.
  • Development Set: A subset of the ESTER2 corpus (about 4.5 hours of audio signal and 36,800 running words) was used for tuning model parameters.
  • Test Set: The remaining portion of the ESTER2 corpus (about 5.9 hours of audio signal and 63,000 running words) was used for final validation.

Key Insights and Ideas

  • Boundary uncertainty is a real problem: The paper convincingly demonstrates that errors in speech segment boundaries directly impact speaking rate estimation and, consequently, ASR performance.
  • Smoothing the speaking rate: By distributing counts across adjacent speaking rate bins during training, the authors effectively smooth the distributions, which is beneficial for less represented speaking rate categories.
  • Probabilistic decoding: Incorporating boundary uncertainty probabilistically during decoding allows the system to be more robust to variations in speaking rate estimation.
  • Speaking rate matters: The results consistently show that modeling speaking rate, even without explicitly handling boundary uncertainty, provides an improvement over simpler models.

Key Results

The paper reports consistent improvements in Word Error Rate (WER) by incorporating speaking rate dependent modeling and handling boundary uncertainty.

  • Speaking rate dependent modeling alone: Provided an improvement over uniform probability and frequency-based estimates.
  • Handling boundary uncertainty in training: Led to a slight improvement in WER (e.g., from 21.95% to 21.83% on the development set when combined with speaking rate dependent modeling).
  • Handling boundary uncertainty in decoding: Also showed a slight improvement (e.g., from 21.95% to 21.95% on the development set when combined with speaking rate dependent modeling).
  • Combining both training and decoding uncertainty handling: Achieved the best results, showing a marginal but consistent improvement over baseline speaking rate dependent modeling (e.g., from 21.95% to 21.85% on the development set).
  • Window size impact: Using a larger window (9 words) for speaking rate estimation yielded better results than a smaller window (5 words).

Limitations and Future Work

  • Arbitrary weighting: The paper acknowledges that the weights used for combining probabilities in the decoding phase are currently arbitrary and could be refined.
  • Limited optimization: The parameter $\delta$ (representing boundary uncertainty) was not extensively optimized.
  • Future Work: The authors suggest extending this approach to other types of variability sources beyond speaking rate, as the demonstrated benefits are promising. They also aim to refine the weighting schemes and optimize parameters.

Summary updated

BibTeX
@inproceedings{jouvet2011about,
  author = {Denis Jouvet and Dominique Fohr and Irina Illina},
  title = {{About handling boundary uncertainty in a speaking rate dependent modeling approach}},
  booktitle = {Interspeech},
  year = {2011},
}
Figure