Factored MLLR Adaptation Algorithm for HMM-based Expressive TTS

June Sig Sung, Doo Hwa Hong, Hyun Woo Koo, Nam Soo KimInterspeech 2012 · OpenReview

Abstract

One of the most popular approaches to parameter adaptation in hidden Markov model (HMM) based systems is the maximum likelihood linear regression (MLLR) technique. In our previous work, we proposed factored MLLR (FMLLR) where an MLLR parameter is defined as a function of a control parameter vector. We presented a method to train the FMLLR parameters based on a general framework of the expectation-maximization (EM) algorithm. To show the effectiveness, we applied the FMLLR to adapt the spectral envelope feature of the reading-style speech to those of the singing voice. In this paper, we apply the FMLLR to the HMM-based expressive speech synthesis task and compare its performance with conventional approaches. In a series of experimental results, the FMLLR shows better performance than conventional methods.

Introduction

Creating high-quality Text-to-Speech (TTS) systems that convey emotion is a significant challenge due to the "data hunger" problem. Training a model to sound angry or joyful typically requires hours of high-quality recordings for each emotional state, which is both expensive and time-consuming. In HMM-based synthesis, researchers often use adaptation techniques to "nudge" a neutral voice model toward an emotional style using limited expressive data. While Maximum Likelihood Linear Regression (MLLR) is the standard tool, it can be too rigid to capture emotional nuances from small datasets.

The Factored MLLR (FMLLR) Approach

The authors propose Factored MLLR (FMLLR) as a more flexible alternative. In standard MLLR, a linear transformation ($y = Mx + b$) is applied to the neutral voice parameters. In contrast, FMLLR makes the transformation matrices themselves a function of a "control vector" representing emotional intensity.

By "factoring" the transformation, each element in the matrices is calculated as an inner product between a learned regression vector and a transformed control vector. This compact framework allows the system to model various emotional styles and intensities with fewer parameters, making it robust even when only a few minutes of emotional audio are available. The parameters are optimized using the Expectation-Maximization (EM) algorithm.

Experimental Setup

The system was evaluated using Korean speech data:

  • Base Models: Neutral models trained on 4,000 utterances (~8-9 hours) from one male and one female speaker.
  • Adaptation Data: 200 utterances (~21 minutes) per emotion (Angry, Sad, Joyful, Fearful) from different male and female speakers.

Results and Discussion

Objective Evaluation

The researchers compared FMLLR against Expression Dependent Models (EDM) and Multi-Representative HMM (MRHSMM) using cepstral distance. Lower values indicate a closer match to human speech.

Table 1: Objective evaluation using average cepstral distance across different emotions and methods.

As shown in the table, FMLLR consistently outperformed MRHSMM. Notably, it also performed better than the "matched" EDM models in categories like "Fearful" and "Sad," despite EDM being specifically trained for those emotions.

Subjective Evaluation

A listening test with 20 evaluators assessed the preference for FMLLR-generated speech compared to the other methods. Positive scores indicate a preference for the FMLLR method.

Figure 4: Subjective preference scores for male (a) and female (b) voices across four emotions.

The results indicate that listeners generally preferred FMLLR over both EDM and MRHSMM across almost all emotional categories for both male and female voices. The "Angry" and "Fearful" categories showed particularly strong preferences for the FMLLR approach.

Conclusion

FMLLR provides a powerful bridge between traditional adaptation and style modeling. Its parameter efficiency makes it ideal for expressive TTS where data is scarce. Future work will explore applying FMLLR to prosodic features like pitch and duration, as well as testing its effectiveness across different languages.

Summary updated

Abstract

One of the most popular approaches to parameter adaptation in hidden Markov model (HMM) based systems is the maximum likelihood linear regression (MLLR) technique. In our previous work, we proposed factored MLLR (FMLLR) where an MLLR parameter is defined as a function of a control parameter vector. We presented a method to train the FMLLR parameters based on a general framework of the expectation-maximization (EM) algorithm. To show the effectiveness, we applied the FMLLR to adapt the spectral envelope feature of the reading-style speech to those of the singing voice. In this paper, we apply the FMLLR to the HMM-based expressive speech synthesis task and compare its performance with conventional approaches. In a series of experimental results, the FMLLR shows better performance than conventional methods.
BibTeX
@inproceedings{sung2012factored,
  author = {June Sig Sung and Doo Hwa Hong and Hyun Woo Koo and Nam Soo Kim},
  title = {{Factored MLLR Adaptation Algorithm for HMM-based Expressive TTS}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=89DgeKGIrD},
}
Figure