METTS: Multilingual Emotional Text-to-Speech by Cross-speaker and Cross-lingual Emotion Transfer

Xinfa Zhu, Yi Lei, Tao Li, Yongmao Zhang, Hongbin Zhou, Heng Lu, Lei XieIEEE/ACM Trans. ASLP 2023 · arXiv

TL;DR

This paper introduces METTS, a novel multilingual emotional Text-to-Speech (TTS) system that enables cross-speaker and cross-lingual emotion transfer. METTS addresses the challenges of foreign accents and speech attribute entanglement by employing multi-scale emotion modeling and information perturbation, achieving more natural and diverse emotional speech synthesis.

What Problem Does This Paper Address?

Modern Text-to-Speech (TTS) systems have made significant strides in naturalness, but achieving diversity and controllability, particularly in multilingual and emotional speech synthesis, remains a challenge. Existing multilingual TTS approaches often leverage monolingual data for cross-lingual synthesis but struggle with emotional expression. Synthesizing emotional speech across different languages and speakers is particularly difficult due to the complex interplay of speaker timbre, emotion, and language in speech signals. This entanglement can lead to foreign accents and speaker timbre leakage, diminishing the quality of the synthetic speech. Furthermore, data efficiency is crucial, as high-quality annotated data is often scarce. This paper tackles the problem of creating multilingual emotional speech that is both natural-sounding and emotionally expressive, even when speakers only have neutral speech data available in their native language.

Methods

METTS builds upon the DelightfulTTS framework and introduces several key innovations:

  • Multi-scale Emotion Modeling: To disentangle language-agnostic and language-specific emotional cues, METTS models emotion at two scales. A Global Style Token (GST) layer captures coarse-grained, language-agnostic prosody patterns (e.g., general pitch changes for anger), while a Conditional Variational Autoencoder (CVAE) models fine-grained, language-specific prosody variations (e.g., subtle nuances in pronunciation). This approach aims to transfer emotions without introducing foreign accents.
  • Information Perturbation for Speaker Disentanglement: To decouple speaker timbre from emotion and language, METTS applies a formant shift to the reference speech signal. This perturbation helps create a speaker-independent emotional representation, preventing speaker timbre leakage during cross-speaker transfer.
  • VQ-based Emotion Matcher: For diverse emotional expression and to address the challenge of selecting appropriate reference speech during inference, METTS employs a Vector Quantization (VQ) based emotion matcher. This module quantizes coarse-grained emotion representations into a reference pool and then uses a classifier to match the input text with the most suitable reference, simplifying the complex regression task and enabling reference-free inference.

METTS offers two modes: METTS-REF for emotion transfer from a reference mel-spectrogram and METTS-ID for emotion synthesis guided by an emotion ID and the emotion matcher.

Datasets

The experiments were conducted on Chinese and English datasets.

  • Chinese Dataset: Included audio clips from two female speakers (CN1, CN2) expressing six emotions (anger, fear, happiness, sadness, surprise, neutral). Totaling 22,205 audio clips (approx. 21 hours).
  • English Dataset: Included audio clips from two female speakers (EN1, EN2) with no apparent emotional expression (i.e., primarily neutral speech). Totaling 19,676 audio clips (approx. 20 hours).

All data were studio-quality recorded at 48KHz.

Key Insights and Ideas

  • Decomposing Emotion for Cross-Lingual Synthesis: The core idea that emotional expression can be broken down into language-agnostic (coarse-grained) and language-specific (fine-grained) components is crucial for overcoming foreign accent issues in cross-lingual emotional TTS.
  • Perturbation as a Tool for Disentanglement: Using simple signal perturbation (formant shift) to effectively remove speaker timbre information from reference speech is an elegant way to achieve speaker independence without complex adversarial training.
  • VQ for Efficient Reference Selection: Transforming the complex task of reference selection into a classification problem via VQ significantly simplifies the modeling and enables more diverse emotional outputs.
  • The Challenge of English Neutral Data: The study highlights that while Chinese data included emotional expressions, the English data was primarily neutral, posing a significant challenge for cross-lingual emotional synthesis.

Key Results

  • Subjective Evaluation: METTS consistently outperformed baseline models (CET, M3) in naturalness, speaker similarity, and emotion similarity for both Chinese and English speakers. METTS-REF achieved the highest speaker and emotion similarity, while METTS-ID demonstrated strong naturalness.
  • Objective Evaluation: METTS achieved higher speaker cosine similarity and lower character/word error rates (CER/WER) compared to baselines, indicating effective speaker disentanglement and more accurate pronunciation.
  • Ablation Studies: Removing GST significantly impacted emotion similarity, highlighting the importance of the language-agnostic representation. Removing CVAE led to a sharp decline in naturalness and emotion similarity, emphasizing the role of fine-grained emotion modeling. Omitting the perturbation module compromised naturalness and speaker similarity, underscoring the need for speaker disentanglement.
  • Emotion Embedding Visualization (T-SNE): Visualizations showed clear clusters for Chinese emotions, validating the emotion classifier. The intermixing of English and Chinese emotion embeddings suggested the language-agnostic representation's ability to capture subtle emotional expressions across languages. Speaker-colored visualizations indicated speaker independence in the emotion embeddings.

Limitations and Future Work

  • English Corpus Limitation: The study acknowledges that the English training corpus was primarily neutral. Leveraging emotional English corpus for training is suggested as a way to improve the expressiveness of English synthetic speech in multilingual emotional TTS.
  • Trade-off in VQ: The paper notes a trade-off between the diversity of emotion embeddings and the predicted accuracy of the emotion matcher as the number of clusters (N) increases. The optimal value of N was found to be 64.
  • Further Improvement for English: While METTS showed good performance, the paper suggests that further improvements are needed for synthesizing English emotional speech for both Chinese and English speakers, likely by addressing the neutral English data limitation.

Summary updated

Abstract

Previous multilingual text-to-speech (TTS) approaches have considered leveraging monolingual speaker data to enable cross-lingual speech synthesis. However, such data-efficient approaches have ignored synthesizing emotional aspects of speech due to the challenges of cross-speaker cross-lingual emotion transfer - the heavy entanglement of speaker timbre, emotion, and language factors in the speech signal will make a system produce cross-lingual synthetic speech with an undesired foreign accent and weak emotion expressiveness. This paper proposes the Multilingual Emotional TTS (METTS) model to mitigate these problems, realizing both cross-speaker and cross-lingual emotion transfer. Specifically, METTS takes DelightfulTTS as the backbone model and proposes the following designs. First, to alleviate the foreign accent problem, METTS introduces multi-scale emotion modeling to disentangle speech prosody into coarse-grained and fine-grained scales, producing language-agnostic and language-specific emotion representations, respectively. Second, as a pre-processing step, formant shift-based information perturbation is applied to the reference signal for better disentanglement of speaker timbre in the speech. Third, a vector quantization-based emotion matcher is designed for reference selection, leading to decent naturalness and emotion diversity in cross-lingual synthetic speech. Experiments demonstrate the good design of METTS.
BibTeX
@article{zhu2023metts,
  author = {Xinfa Zhu and Yi Lei and Tao Li and Yongmao Zhang and Hongbin Zhou and Heng Lu and Lei Xie},
  title = {{METTS: Multilingual Emotional Text-to-Speech by Cross-speaker and Cross-lingual Emotion Transfer}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2023},
}
Figure