Speaker and language adaptive training for HMM-based polyglot speech synthesis

Heiga ZenInterspeech 2010 · DOI

Introduction

Developing a single speech synthesizer capable of speaking multiple languages with high naturalness—a "polyglot" system—presents significant technical challenges. Traditional HMM-based (Hidden Markov Model) synthesizers often struggle with acoustic variability when mixing data from diverse languages and speakers. This variability typically results in "blurry" models that fail to distinguish between language-specific phonetic nuances and individual speaker characteristics. Furthermore, capturing the unique prosodic and phonetic rules of different languages using a single decision tree is often insufficient. This paper introduces SLAT (Speaker and Language Adaptive Training), a framework designed to simultaneously adapt to both language structure and speaker identity.

The SLAT Framework

The proposed SLAT model integrates two powerful adaptive training techniques to handle the complexities of multilingual speech. First, it employs Cluster Adaptive Training (CAT) for language adaptation. In this approach, the mean vectors of the model are represented as a linear combination of "prototype" language clusters. Unlike standard methods that use a single universal decision tree, SLAT utilizes cluster-dependent decision trees, allowing each language or cluster to maintain its own unique branching logic for context-dependency.

Second, the system incorporates Constrained Maximum Likelihood Linear Regression (CMLLR) to handle speaker adaptation. This applies a feature-space transform to the language-adapted models, effectively mapping the canonical language prototypes to a specific speaker's acoustic profile.

Schematic diagram of the Speaker and Language Adaptive Training (SLAT) framework showing the progression from decision trees to adapted model sets.

To manage the computational cost of building multiple trees across languages, the authors utilize an iterative Expectation-Maximization (EM) approach. This process alternates between estimating the CMLLR transform parameters and updating the canonical model parameters, ensuring an efficient training pipeline despite the high dimensionality of the underlying matrices.

Experimental Setup

The researchers evaluated the SLAT framework using a multilingual database containing five languages: North American English, British English, European Spanish, European French, and Standard German. The dataset included 50 non-professional speakers (10 per language), providing a total of 6,100 utterances for training. This diverse dataset allowed the authors to test the model's ability to generalize across different linguistic structures and vocal characteristics.

One key insight from the training process involved the specificity of pitch ($log F_0$). While spectral features and durations shared many commonalities across languages, pitch was found to be highly language-specific. The model required significantly more leaf nodes for pitch clusters than for other parameters, confirming that prosody is a primary area where languages diverge.

Results and Discussion

The performance of the SLAT model was compared against two baseline models: a Language-Independent (LI-SAT) model and a Language-Dependent (LD-SAT) model. Subjective evaluation was conducted using paired-comparison preference tests with human listeners.

Subjective preference test results (%) comparing SLAT against LI-SAT and LD-SAT baselines.

As shown in the results, listeners significantly preferred the speech generated by the SLAT model. When compared to the LI-SAT baseline, SLAT was preferred 37.6% to 24.2%. The preference was even more pronounced against the LD-SAT baseline, where SLAT was preferred 45.6% to 26.7%. These results, statistically significant at the $p < 0.01$ level, demonstrate that the combined adaptive training framework captures the nuances of polyglot speech far more effectively than traditional mixed or separate training methods.

Conclusion

SLAT represents a significant step forward in polyglot speech synthesis by successfully decoupling language-specific nuances from speaker-specific traits. While the iterative tree reconstruction adds computational complexity, the resulting improvements in naturalness and flexibility are substantial. Future work may explore scaling this framework to a broader range of language families and optimizing the number of prototype clusters for global applications.

Summary updated

BibTeX
@inproceedings{zen2010speaker,
  author = {Heiga Zen},
  title = {{Speaker and language adaptive training for HMM-based polyglot speech synthesis}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-172},
}
Figure