Speech factorization for HMM-TTS based on cluster adaptive training

Javier Latorre, Vincent Wan, Mark J. F. Gales, Langzhou Chen, K. K. Chin, Kate Knill, Masami AkamineInterspeech 2012

TL;DR

This paper introduces a novel method for "factorizing" speech into independent speaker and emotion components within HMM-based Text-to-Speech (TTS) systems. This allows for the synthesis of expressive speech with a target speaker's voice, even when only neutral-style training data is available for that speaker, opening doors for more natural and versatile synthetic voices.

What Problem Does This Paper Address?

High-quality speech synthesizers have made significant strides in intelligibility and naturalness, but a key limitation remains: most systems produce speech in a neutral or "reading" style. Achieving expressive speech, which is crucial for applications like e-book readers and dialogue systems, has been a long-standing challenge. Previous approaches often required large, expression-specific databases, were limited to a single speaker's voice, or could only produce discrete emotions. This paper tackles the problem of controlling speaker identity and emotional expression independently, enabling the synthesis of expressive speech for any target voice, even with limited training data.

Methods

The core of the proposed solution lies in Cluster Adaptive Training (CAT), a technique that allows for the factorization of speech characteristics. In this framework, the authors model speaker identity and expressiveness (emotion) using separate sets of clusters. Each speech factor is represented by a vector of cluster weights. By modifying these weights, users can independently control the speaker's voice and the emotional tone of the synthesized speech.

The training process involves iteratively estimating canonical parameters (like mean vectors and covariance matrices) and CAT weights. A key aspect is the initialization strategy: speaker clusters are first trained using neutral data, and then emotion clusters are added and initialized to model speaker-independent differences between neutral and emotional speech. The paper also details how to initialize and structure the decision trees for these clusters, which capture context-dependent variations.

Datasets

The experiments were conducted using a dataset with 4 speakers (two male, two female). Neutral data was available for all speakers. For one male and one female speaker, additional recordings were made in 8 other styles: angry, happy, sad, surprised, fast, slow, informal, and formal. The total dataset comprised 9697 neutral and 11867 expressive sentences. Speech was sampled at 16kHz and parameterized into mel-cepstral coefficients, logF0, and bark-scaled aperiodicity bands.

Key Insights and Ideas

  • Independent Control: The most significant insight is the successful factorization of speaker and emotion, allowing for independent control. This means you can take a neutral voice and make it sound happy, sad, or angry without needing to retrain the entire system or have specific expressive data for that voice.
  • Leveraging Neutral Data: The approach effectively utilizes neutral-style data for target speakers, a common scenario in real-world TTS development. This overcomes the need for extensive, multi-expression recordings for every desired voice.
  • Continuous Expression Space: The use of continuous weight vectors for cluster interpolation allows for a continuous range of expressions, moving beyond discrete emotional categories.
  • Generalizability: The paper suggests that the core concept of CAT factorization could be extended to other HMM-TTS systems, potentially improving expressive synthesis across a wider range of platforms.

Key Results

Subjective experiments were conducted using ABX preference tests.

  • Speaker-Expression AVM vs. CAT: The CAT model was significantly preferred over a baseline speaker-expression Average Voice Model (AVM), particularly for angry sentences.
  • Neutral-Only Speaker Synthesis: The CAT system demonstrated the ability to produce expressive speech with the voice of a neutral-only target speaker.
  • Perceptual Distance: When comparing synthesized speech to real expressions, the CAT model produced results that were "slightly blurred but still acceptable," with the strongest degradation observed for 'angry' vs. 'happy' expressions. This was attributed to the diverse ways speakers performed anger.
  • Prosody Transplant vs. CAT: The CAT model was strongly preferred over a prosody transplant baseline for expressiveness, with no significant difference in similarity to the target speaker.

Limitations and Future Work

The authors acknowledge that while the frame-level CAT model is effective, some supra-segmental aspects of expression (like global variance) are not fully captured. They suggest that future work could involve combining the current CAT model with other supra-segmental models also trained with a CAT factorization to further improve expressive synthesis.

Summary updated

BibTeX
@inproceedings{latorre2012speech,
  author = {Javier Latorre and Vincent Wan and Mark J. F. Gales and Langzhou Chen and K. K. Chin and Kate Knill and Masami Akamine},
  title = {{Speech factorization for HMM-TTS based on cluster adaptive training}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure