Improved generation of fundamental frequency in HMM-based speech synthesis using generation process model

Miaomiao Wang, Miaomiao Wen, Keikichi Hirose, Nobuaki MinematsuInterspeech 2010

TL;DR

This paper tackles the challenge of generating natural-sounding fundamental frequency (F0) contours in HMM-based speech synthesis, particularly for Mandarin. By integrating a "generation process model" with prior linguistic knowledge about phoneme voicing, the authors propose a method to fix F0 tracking and voiced/unvoiced (VU) decision errors before training HMMs, leading to more accurate and natural synthesized speech.

What Problem Does This Paper Address?

HMM-based Text-to-Speech (TTS) systems are powerful but can suffer from degraded speech quality when the acoustic features used for training are noisy. A major culprit is inaccurate F0 tracking and incorrect voiced/unvoiced (VU) decisions, especially in tonal languages like Mandarin where certain tones (e.g., Tone 3 and Tone 4) can have low and unstable pitch, leading to tracking errors. Traditional HMM approaches struggle with the discontinuous nature of F0 between voiced and unvoiced segments. While Multi-Space Distribution HMMs (MSD-HMMs) offer a solution by modeling these separately, they introduce their own limitations, making it harder to apply advanced techniques like adaptation and potentially leading to over-smoothing or hard assignments that hinder learning. This paper aims to overcome these limitations by proposing a more robust F0 generation method.

Methods

The core of the proposed method is the application of a generation process model for F0 contours. This model, based on physiological properties of the vocal folds, represents F0 as a superposition of "phrase components" (global sentence shape) and "tone components" (local pitch changes), controlled by linguistic information.

Key aspects of their approach include:

  • Pre-training Error Correction: Instead of trying to fix errors during HMM training or synthesis, this method re-estimates F0 contours using the generation process model before HMM training. This effectively corrects pitch tracking errors and interpolates F0 in unvoiced regions.
  • Linguistic Prior for VU Decisions: Prior knowledge about the voicing characteristics of Mandarin phonemes (initials and finals) is used to make more accurate VU decisions. This is crucial because Mandarin syllables have a relatively simple structure (consonant-vowel or vowel), limiting the number of VU switches within a syllable.
  • Single F0 Stream: By generating a continuous F0 contour, the model allows for a single F0 stream in the HMM, simplifying the modeling process and avoiding the complexities of MSD-HMMs.
  • Standard HMM Framework: The corrected F0 contours are then used to train standard HMMs, where each state is modeled with a single Gaussian distribution.

Datasets

The authors used a manually checked female speaker's corpus from the University of Science and Technology of China. This corpus consists of 270 training sentences and 30 testing sentences. The text includes pronunciation and prosodic boundary information.

Key Insights and Ideas

  • Fixing Errors Before Training is Key: The most significant insight is that addressing F0 tracking and VU decision errors prior to HMM training, rather than trying to compensate for them during synthesis, leads to a more fundamental improvement in speech quality.
  • Linguistic Knowledge Enhances Prosody: Leveraging phoneme-level voicing information for Mandarin provides a strong prior for accurate VU decisions, which are critical for natural prosody.
  • Generation Process Model for Smoothness and Interpolation: The generation process model's ability to represent F0 as a combination of phrase and tone components allows for smoother contours and provides a principled way to interpolate F0 in unvoiced regions, a significant challenge for traditional HMMs.
  • Simplifying HMM Modeling: By moving to a single F0 stream and correcting errors upfront, the complexity of HMM modeling (e.g., avoiding MSD-HMM issues) is reduced, potentially enabling easier application of techniques like speaker adaptation.

Key Results

The paper demonstrates that their proposed method outperforms MSD-HMMs in synthesizing Mandarin speech, particularly for challenging syllables like "zhe3" and "le0" which are difficult due to their pitch contour characteristics. Qualitative improvements are noted in reducing VU decision errors and achieving more accurate F0 contours compared to the MSD-HMM approach. While specific quantitative metrics like Mean Opinion Scores (MOS) are not explicitly detailed, the figures illustrate significant improvements in F0 contour accuracy and smoothness. The authors report that almost 22.37% of syllables in their initial analysis had VU decision errors, with a significant portion occurring in T3 and T4 tones, which their method aims to rectify.

Limitations and Future Work

  • Over-smoothing: While the generation process model aims to improve smoothness, the authors acknowledge that synthesized speech can still sound "muffled" compared to natural speech due to over-smoothing in acoustic parameter trajectories. They suggest that the generation process model offers a potential recovery mechanism for prosodic features like tone emphasis and phrase component refinement at the synthesis stage.
  • Future Work: The paper implies that further refinement of the generation process model and its integration with advanced acoustic modeling techniques could lead to even more natural synthesized speech. The ability to flexibly control prosodic features is highlighted as a key advantage for future research.

Summary updated

BibTeX
@inproceedings{wang2010improved,
  author = {Miaomiao Wang and Miaomiao Wen and Keikichi Hirose and Nobuaki Minematsu},
  title = {{Improved generation of fundamental frequency in HMM-based speech synthesis using generation process model}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure