A Bayesian approach to Hidden Semi-Markov Model based speech synthesis

Kei Hashimoto, Yoshihiko Nankaku, Keiichi TokudaInterspeech 2009

TL;DR

This paper introduces a Bayesian approach to Hidden Semi-Markov Model (HSMM) based speech synthesis. By treating model parameters as random variables and incorporating explicit state duration modeling within the HSMM framework, the authors achieve more natural-sounding synthesized speech compared to traditional HMM-based methods.

What Problem Does This Paper Address?

Traditional HMM-based speech synthesis models speech parameters like spectrum and excitation using HMMs. However, a key limitation is that HMMs don't explicitly model the duration a state is occupied, which is crucial for speech rhythm and tempo. While previous methods tried to address this by modeling state durations separately, this created an inconsistency between how models were trained (without explicit duration modeling) and how speech was synthesized (with it). This paper tackles this inconsistency by proposing a unified Bayesian framework that integrates explicit state duration modeling directly into the HSMM, applied to both training and synthesis.

Methods

The core of this paper is the application of a Bayesian approach to Hidden Semi-Markov Models (HSMMs) for speech synthesis.

  • Bayesian Approach: Instead of estimating fixed model parameters (like in Maximum Likelihood - ML), the Bayesian approach treats parameters as random variables with probability distributions. This allows for more robust estimation, especially with limited data, and naturally incorporates uncertainty. The synthesis process then relies on a predictive distribution derived from these parameter distributions.
  • Hidden Semi-Markov Model (HSMM): An HSMM is an extension of the HMM that explicitly models the duration a state can be occupied. This is achieved by incorporating state duration probability distributions directly into the model.
  • Variational Bayesian Method: To handle the computational complexity of exact Bayesian inference, the authors employ the variational Bayesian method. This technique approximates the true posterior distribution of model parameters with a simpler, tractable distribution, allowing for efficient optimization.
  • Unified Framework: The key novelty is applying the Bayesian approach to HSMMs for both training and synthesis. This resolves the inconsistency found in prior work where HMMs were trained without explicit duration modeling but synthesis relied on it. The Bayesian framework allows for a single, consistent predictive distribution to govern the entire synthesis process.
  • Speech Parameter Generation: The paper also details how speech parameters (spectrum and pitch) are generated using the trained HSMMs, leveraging the expectation values of the model parameters derived from the variational Bayesian inference.

Datasets

  • ATR Japanese speech database (B-set): This database, consisting of 503 phonetically balanced sentences uttered by a male speaker, was used for training and evaluation. The first 450 sentences were used for training, and the remaining 53 for evaluation.

Key Insights and Ideas

  • Bridging the Training-Synthesis Gap: The most significant insight is that by using HSMMs within a Bayesian framework, the paper successfully unifies the modeling of state durations for both training and synthesis. This eliminates a critical inconsistency in previous HMM-based systems.
  • Bayesian Approach for Robustness: The Bayesian approach, by treating parameters as distributions, inherently handles uncertainty and mitigates over-fitting, especially when dealing with limited training data. This is crucial for achieving natural-sounding speech.
  • HSMMs Enhance Naturalness: Explicitly modeling state durations with HSMMs, when integrated into the Bayesian framework, leads to demonstrably more natural synthesized speech compared to models that don't explicitly handle duration.
  • Model Structure Selection: The Bayesian approach offers a principled way to select model structure (e.g., number of states) by maximizing marginal likelihood, avoiding asymptotic assumptions of criteria like MDL, which can be problematic with small datasets.

Key Results

  • Improved Naturalness: Subjective listening tests (MOS scores) showed that the proposed "Bayes-HSMM" model achieved significantly better naturalness scores than the conventional "Bayes-HMM" and "ML-HSMM" models.
  • HSMM Advantage: The "ML-HSMM" also outperformed the "ML-HMM," indicating the benefit of explicit duration modeling even within a non-Bayesian framework.
  • Bayesian Advantage: The "Bayes-HSMM" outperformed the "ML-HSMM," highlighting the benefits of the Bayesian approach in overcoming over-fitting and over-smoothing issues, particularly with larger model structures.
  • Larger Model Structures: The Bayesian models ("Bayes-HMM" and "Bayes-HSMM") used considerably larger model structures (more states) than their ML counterparts, which the authors attribute to the Bayesian approach's ability to avoid over-fitting and over-smoothing.

Limitations and Future Work

  • Computational Cost: While the variational Bayesian method makes the approach tractable, the paper acknowledges that the computational cost is increased compared to ML criteria, especially with larger model structures.
  • Future Work: The authors suggest investigating the relationship between speech quality and the size of the model structure further.

Summary updated

BibTeX
@inproceedings{hashimoto2009bayesian,
  author = {Kei Hashimoto and Yoshihiko Nankaku and Keiichi Tokuda},
  title = {{A Bayesian approach to Hidden Semi-Markov Model based speech synthesis}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure