A new approach of speaking rate modeling for Mandarin speech prosody

Chiao-Hua Hsieh, Chen-Yu Chiang, Yih-Ru Wang, Hsiu-Min Yu, Sin-Horng ChenInterspeech 2012

TL;DR

This paper introduces a novel method for modeling Mandarin speech prosody by treating speaking rate as a continuous variable. By adapting a hierarchical prosodic model (HPM) to account for speaking rate, the authors provide a more systematic way to understand and quantify its impact on speech features, paving the way for improved speech synthesis and recognition systems.

What Problem Does This Paper Address?

Speech prosody, encompassing rhythm, intonation, and stress, is crucial for natural-sounding speech. Speaking rate – how fast or slow someone speaks – significantly influences prosodic features like syllable duration, pause frequency, and pitch contours. Existing methods often model speaking rate discretely (e.g., fast, normal, slow) or struggle to capture its continuous impact. This makes it challenging for applications like Text-to-Speech (TTS) to generate natural-sounding speech at varying speeds or for Automatic Speech Recognition (ASR) to handle the variability introduced by different speaking rates. The authors aim to bridge this gap by developing a model that can continuously account for speaking rate's influence on Mandarin speech prosody.

Methods

The core of the proposed approach is a modification of their previously developed Hierarchical Prosodic Model (HPM) [18]. The HPM models the complex relationships between prosodic-acoustic features, prosodic structure, and linguistic features. The key innovation here is treating speaking rate (SR) as a continuous independent variable.

  1. SR Estimation: Speaking rate is calculated for each utterance as the average syllable duration, excluding pauses.
  2. Prosodic-Acoustic Feature Normalization: To mitigate the direct impact of SR on acoustic features, they employ a novel normalization strategy. Instead of simple Z-normalization, they use SR-specific normalization functions. These functions are derived by fitting smooth curves (polynomials) to the relationship between utterance-wise means/standard deviations of features (like syllable duration, pause duration, LogF0 contour coefficients) and their corresponding speaking rates. This approach aims to preserve variations due to factors other than speaking rate.
  3. Modified PLM Algorithm: A modified version of their Joint Prosody Labeling and Modeling (PLM) algorithm is used. The crucial change is making the break-syntax submodel SR-dependent. This means the probability of certain prosodic breaks (like pauses) occurring is modeled as a function of speaking rate, acknowledging that pause frequency is highly sensitive to speaking speed.

Datasets

The study utilizes four parallel corpora of a single female speaker, recorded at fast, normal, medium, and slow speaking rates. These corpora consist of short paragraphs, totaling 1,478 utterances and 203,746 syllables. The paper does not introduce a new benchmark dataset but uses existing data to validate their approach.

Key Insights and Ideas

  • Speaking Rate as a Continuous Variable: Treating speaking rate as a continuous variable, rather than discrete categories, allows for a more nuanced and accurate modeling of its effects on prosody.
  • SR-Specific Normalization: The proposed method of fitting smooth curves to the relationship between feature statistics (mean, std dev) and speaking rate is a more sophisticated way to normalize acoustic features compared to traditional Z-normalization. It aims to disentangle the effects of speaking rate from other sources of variation.
  • Break-Syntax SR-Dependence: The finding that the occurrence probability of certain break types (like pauses) is directly dependent on speaking rate is crucial. The model explicitly incorporates this, leading to more linguistically meaningful prosodic labeling.
  • Linguistic Meaningfulness: The authors emphasize that both the trained HPM parameters and the labeled prosodic tags are linguistically meaningful, suggesting the model captures genuine aspects of speech production.

Key Results

  • The analysis of speaking rate's effects on model parameters showed good agreement with prior linguistic knowledge.
  • The normalization method effectively compensated for the effects of speaking rate on prosodic-acoustic features, particularly for pause durations in inter-word junctures.
  • The occurrence probability of certain break types, like the short-pause minor break (B2-2), was found to be higher in slower speech, and the occurrence probability of major breaks (B4) also showed a dependency on speaking rate, aligning with expectations.
  • The modified PLM algorithm converged in 94 iterations, indicating computational feasibility.

Limitations and Future Work

  • Single Speaker: The current model is trained on data from a single female speaker. The authors acknowledge that further work is needed to generalize the model to multiple speakers.
  • Future Applications: The authors plan to apply this modeling approach to construct HPMs for assisting in ASR and for speaking rate conversion in TTS systems. This suggests future work will focus on integrating the model into practical speech technology applications.

Summary updated

BibTeX
@inproceedings{hsieh2012new,
  author = {Chiao-Hua Hsieh and Chen-Yu Chiang and Yih-Ru Wang and Hsiu-Min Yu and Sin-Horng Chen},
  title = {{A new approach of speaking rate modeling for Mandarin speech prosody}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure