Discontinuous observation HMM for prosodic-event-based F0 generation

Tomoki Koriyama, Takashi Nose, Takao KobayashiInterspeech 2012

TL;DR

This paper introduces a novel "Discontinuous Observation HMM" (DO-HMM) to better model and generate fundamental frequency (F0) contours in spontaneous speech synthesis. By treating unvoiced speech segments as "missing data" rather than interpolating them, the DO-HMM achieves more compact models and comparable or improved F0 generation quality compared to existing methods, particularly for prosodic-unit based HMMs.

What Problem Does This Paper Address?

Generating natural-sounding speech, especially spontaneous speech, requires accurately modeling prosody, particularly the fundamental frequency (F0) contour. Traditional HMM-based speech synthesis often relies on phone-unit HMMs, which can be parameter-heavy. The authors previously proposed a "prosodic-unit HMM" that uses segments between prosodic events (like pitch accents) as modeling units, leading to more compact F0 models. However, real speech often has discontinuous F0 observations within these prosodic units due to unvoiced segments. Existing methods like MSD-HMM and continuous F0 HMM struggle to effectively model these discontinuities without potentially compromising accuracy through interpolation or complex weighting schemes. This paper aims to bridge this gap by proposing a method that directly handles discontinuous F0 data within the prosodic-unit HMM framework.

Methods

The core contribution is the Discontinuous Observation HMM (DO-HMM). The key idea is to treat unvoiced frames as "missing data" in the F0 sequence. This allows the model to optimize its parameters using only the observed voiced F0 data, avoiding the potential inaccuracies introduced by interpolating unvoiced segments.

The DO-HMM is trained using an Expectation-Maximization (EM) algorithm, specifically adapted to handle missing data. The Q-function is decomposed to optimize state transition probabilities and output probability density functions (pdfs). The update equations for the output pdfs are weighted sums of observed values and previous parameters, aiming for robust estimation.

Additionally, the paper explores two F0 generation methods:

  1. Whole region generation: Maximizes the likelihood of the entire F0 sequence (including interpolated unvoiced regions) and then uses the voiced segments for synthesis.
  2. Voiced region generation: Maximizes the likelihood of only the observed voiced F0 segments, relying on phone-unit HMMs for voiced/unvoiced information.

The authors also employ decision tree-based context clustering to further optimize the DO-HMM, grouping similar contexts to create more compact models.

Datasets

The experiments were conducted using spontaneous speech data from the Corpus of Spontaneous Japanese (CSJ). This dataset includes rich prosodic labels, specifically X-JTOBI tone tier labels, which are crucial for defining prosodic events and units. Data from two non-professional female speakers was used, with approximately 25 minutes of speech per speaker. The data was divided into lecture and conversation types.

Key Insights and Ideas

  • Missing Data is Key: Treating unvoiced F0 as "missing" rather than interpolating it is a more principled way to handle discontinuous F0 in spontaneous speech synthesis. This avoids relying on potentially inaccurate interpolated values during training.
  • Prosodic Units Benefit from DO-HMM: The DO-HMM is particularly well-suited for prosodic-unit HMMs, which inherently deal with segments that can contain unvoiced portions.
  • Compact Models: The DO-HMM, especially when combined with decision tree clustering, leads to significantly more compact F0 models (fewer leaf nodes in decision trees) compared to MSD-HMM and continuous F0 HMMs. This is crucial for efficient model adaptation and synthesis, especially with limited data.
  • Generation Strategy Matters: While the DO-HMM itself is novel, the choice of generation method (whole region vs. voiced region) impacts the results. Voiced region generation led to higher RMSE, suggesting that smoothing between voiced segments is important for naturalness.

Key Results

  • The DO-HMM achieved the lowest average F0 distortion (RMSE) of 241.8 ms when using the whole region generation method, outperforming MSD-HMM (277.9 ms) and continuous F0 HMM (309.0 ms).
  • The correlation coefficient was comparable across methods for whole region generation, with DO-HMM achieving 0.616.
  • The DO-HMM resulted in the smallest number of leaf nodes in the F0 decision trees (241.8), indicating a more compact and potentially more robust model compared to MSD-HMM (277.9) and continuous F0 HMM (309.0).
  • Voiced region generation generally resulted in higher RMSEs than whole region generation across all training methods.

Limitations and Future Work

  • Smoothing for Voiced Region Generation: The paper notes that voiced region generation produced less smooth F0 contours between voiced segments, leading to higher RMSE. This suggests that post-processing or trajectory modeling might be needed to improve smoothness.
  • Frame-based Modeling: The current HMM framework is frame-based. The authors suggest incorporating trajectory models in future work to better capture the temporal dependencies of F0 observations, potentially enhancing the utility of the missing data approach.
  • Model Adaptation: While the paper highlights the advantage of DO-HMM for model adaptation due to its compactness, further investigation into adaptation scenarios would be valuable.

Summary updated

BibTeX
@inproceedings{koriyama2012discontinuous,
  author = {Tomoki Koriyama and Takashi Nose and Takao Kobayashi},
  title = {{Discontinuous observation HMM for prosodic-event-based F0 generation}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure