Rich context modeling for high quality HMM-based TTS

Zhi-Jie Yan, Yao Qian, Frank K. SoongInterspeech 2009 · OpenReview

Introduction

Hidden Markov Model (HMM) based Text-to-Speech (TTS) systems are valued for their flexibility and efficiency. However, a persistent challenge in the field is the "over-smoothing" problem, where synthesized speech lacks the crispness of natural audio and sounds "muffled" or "buzzy." This occurs because standard systems use decision tree tying to group similar allophones into leaf nodes, averaging their statistical parameters to handle data sparsity. This averaging acts as a low-pass filter, blurring formants and spectral transitions.

Methodology

The authors propose a two-stage approach to mitigate over-smoothing while maintaining model stability through Rich Context Modeling.

Rich Context Training

Instead of relying solely on tied models, the system estimates "rich context models" for every specific context seen in the training data. To ensure numerical stability, the authors apply a parameter-splitting strategy: the means (carrying critical spectral and transition information) are estimated for each specific rich context, while the variances remain tied to the decision tree. This allows the model to capture fine-grained detail without collapsing due to sparse data.

To handle unseen context combinations during synthesis, the system generates a "sausage" (or lattice) of candidate rich context models. The standard tied-model sequence is used as a "global trend" guide. The system then searches the sausage for the rich context sequence that minimizes the distance to this guide, calculated using an upper-bound of the Kullback-Leibler Divergence (KLD) to account for spectrum, pitch, and duration simultaneously.

Experimental Evaluation

The approach was evaluated using a 15-hour American English database from a professional female speaker. Audio was sampled at 16 kHz, with 40th-order Line Spectral Pair (LSP) coefficients used for spectral modeling.

Spectral Analysis

Visualizations of the spectral envelopes reveal that rich context models produce much sharper formant structures and more detailed transitions compared to the "blurry" baseline models. This visual evidence supports the claim that averaging in conventional tied models is a primary cause of quality degradation.

Comparison of spectral envelopes between conventional tied models and rich context models.

Subjective Results

In a preference test conducted with six language experts, the rich context modeling approach was preferred in 76% of cases, compared to only 20% for the baseline tied model. Listeners specifically highlighted improvements in the "crispness" and "clarity" of the output, confirming that the preservation of fine acoustic details significantly enhances perceived quality.

Subjective preference test results showing a strong preference for the rich context model.

Conclusion

Rich context modeling effectively breaks the trade-off between model stability and acoustic detail in HMM-based TTS. By selectively untying means and using a guiding search mechanism, the method significantly improves the perceived quality and clarity of synthesized speech, addressing a long-standing limitation of statistical parametric synthesis.

Summary updated

BibTeX
@inproceedings{yan2009rich,
  author = {Zhi-Jie Yan and Yao Qian and Frank K. Soong},
  title = {{Rich context modeling for high quality HMM-based TTS}},
  booktitle = {Interspeech},
  year = {2009},
  url = {https://openreview.net/forum?id=WAVcUpYZTuU},
}
Figure