An evaluation of parameter generation methods with rich context models in HMM-based speech synthesis

Shinnosuke Takamichi, Tomoki Toda, Yoshinori Shiga, Hisashi Kawai, Sakriani Sakti, Satoshi NakamuraInterspeech 2012

TL;DR

This paper tackles the "muffled" sound in HMM-based speech synthesis by proposing new parameter generation methods that leverage "rich context models." These methods reformulate rich context models as Gaussian Mixture Models (GMMs) and generate speech parameters using a maximum likelihood criterion, significantly improving synthetic speech quality while retaining flexibility.

What Problem Does This Paper Address?

Traditional HMM-based speech synthesis, while offering flexibility in controlling voice characteristics, often suffers from over-smoothing of generated speech parameters, leading to muffled and unnatural-sounding speech. Hybrid methods combining HMMs with unit selection have improved quality by using natural waveform segments, but they often sacrifice the flexibility of HMMs. "Rich context models" were introduced to represent acoustic segments with probability distributions, offering better flexibility than pure unit selection. However, the conventional synthesis method using these models lost the efficient and flexible acoustic modeling capabilities of the original HMM-based approach. This paper aims to bridge this gap by proposing parameter generation methods that utilize rich context models within the HMM framework, seeking to achieve both high quality and flexibility.

Methods

The core of the proposed methods lies in how they represent and utilize "rich context models."

  • Rich Context Modeling: Instead of a single Gaussian distribution per HMM state, rich context models use individual mean vectors for each full context label and tie covariance matrices across labels within a leaf node of a decision tree. This aims to reduce over-smoothing while maintaining robustness.
  • Representation as GMM: The paper proposes reforming the rich context models within each HMM leaf node into a Gaussian Mixture Model (GMM). This GMM represents the probability distribution of acoustic features for that state.
  • Parameter Generation:
    • EM Algorithm: The paper proposes using the Expectation-Maximization (EM) algorithm to generate speech parameters. This involves iteratively updating the static feature vector sequence by maximizing the HMM likelihood under a constraint relating static and dynamic features.
  • Single Mixture Component Sequence: An approximation is also proposed where the HMM likelihood is simplified to consider only a single mixture component sequence. This involves iteratively updating the mixture component sequence and then the static feature vector sequence. This approach is noted to be similar to unit selection.

The key novelty is the integration of rich context models into a GMM framework for parameter generation within the HMM synthesis process, aiming to retain the flexibility of HMMs while mitigating over-smoothing.

Datasets

  • Training Data: 450 sentences from the ATR Japanese speech database.
  • Evaluation Data: 53 sentences from the ATR Japanese speech database.

The paper does not introduce new benchmarks but evaluates against existing methods using standard speech synthesis datasets.

Key Insights and Ideas

  • GMMs as a Bridge: Representing rich context models as GMMs within HMM states provides a flexible way to model acoustic features, combining the strengths of both statistical parametric and unit selection approaches.
  • Likelihood vs. Target: The paper highlights a crucial finding: while the likelihood of generated speech parameters increases during iterative refinement, the likelihood of natural speech parameters is highly dependent on the initial parameter sequence and doesn't always improve. This suggests that likelihood alone might not be the best metric for selecting optimal rich context models.
  • Approximation Matters: The "single mixture component sequence" approximation, while simpler, yielded better speech quality than the EM algorithm in their experiments, suggesting that a more direct selection of acoustic segments (akin to unit selection) can be effective.
  • Frame vs. State Selection: Both frame-based and state-based selection units for rich context models proved effective, with no significant difference observed in their impact on speech quality.

Key Results

  • Significant Quality Improvement: The proposed methods, both the EM-based and the single mixture component sequence approach, demonstrated significant improvements in the quality of synthetic speech compared to the conventional HMM-based method.
  • "Proposed(single)" Outperforms EM: The single mixture component sequence method ("Proposed(single)") generally yielded better speech quality than the EM algorithm-based method ("Proposed(GMM)").
  • "Proposed(target)" Best: The "Proposed(target)" method, which used natural speech parameters as a reference for selecting rich context models, achieved the highest preference scores, indicating its effectiveness, though it's noted as difficult to select using likelihood alone.
  • Full Synthesis Effectiveness: The proposed methods also showed significant quality improvements when used in a full synthesis process, generating all speech parameters.

Limitations and Future Work

  • Likelihood as a Selection Measure: The paper explicitly states that suitable rich context models are not effectively selected using the likelihood measure alone. This is a significant limitation.
  • Improvement of Selection Measure: As a direct consequence of the above, the authors identify improving the selection measure for rich context models as a key area for future work.
  • Difference in Full Synthesis: The difference between "Proposed(single)" and "Proposed(target)" was larger in full synthesis than in spectral component evaluation. The authors plan to investigate the causes of this difference in future work.
  • Global Variance: The paper mentions that global variance was not considered in their experiments, suggesting its potential inclusion as a future enhancement.

Summary updated

BibTeX
@inproceedings{takamichi2012evaluation,
  author = {Shinnosuke Takamichi and Tomoki Toda and Yoshinori Shiga and Hisashi Kawai and Sakriani Sakti and Satoshi Nakamura},
  title = {{An evaluation of parameter generation methods with rich context models in HMM-based speech synthesis}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure