DSE-TTS: Dual Speaker Embedding for Cross-Lingual Text-to-Speech

Sen Liu, Yiwei Guo, Chenpeng Du, Xie Chen, Kai Yu

Abstract

Although high-fidelity speech can be obtained for intralingual speech synthesis, cross-lingual text-to-speech (CTTS) remains a significant challenge. The primary difficulty lies in accurately retaining speaker timbre (identity) while simultaneously eliminating accents from the speaker's native language to achieve "nativeness." In this paper, we demonstrate that vector-quantized (VQ) acoustic features contain significantly less speaker information than traditional mel-spectrograms. Leveraging this finding, we propose a novel Dual Speaker Embedding TTS (DSE-TTS) framework. By decoupling the linguistic speaking style from the vocal timbre—assigning the former to the acoustic model and the latter to the vocoder—DSE-TTS achieves authentic cross-lingual synthesis that significantly outperforms current state-of-the-art models.

Introduction

Cross-lingual text-to-speech (CTTS) aims to synthesize speech in a target language using the voice of a speaker who may not speak that language. The ideal CTTS system achieves two conflicting goals: vocal identity preservation (the voice must sound like the target person) and nativeness (the speech must have the correct accent and rhythm of the target language).

The core obstacle in CTTS is "entanglement." Traditional TTS models typically rely on mel-spectrograms as intermediate representations. However, mel-spectrograms tightly couple linguistic content, prosody, and speaker identity. When transferring a voice to a new language, this entanglement often causes the model to either carry over the speaker's original accent or lose their unique vocal characteristics in an attempt to sound native.

The DSE-TTS Framework

1. Speaker-Independent VQ Features

To solve the entanglement problem, the researchers investigated the use of discrete vector-quantized (VQ) acoustic features, specifically those derived from wav2vec 2.0. Through speaker classification experiments, the authors found that while a classifier could identify a speaker from a mel-spectrogram with ~70% accuracy, that accuracy dropped to ~20% when using VQ features. This empirical evidence suggests that VQ features serve as a "neutral" bridge, focusing on linguistic content while filtering out speaker-specific traits.

2. Dual Speaker Embedding (DSE) Strategy

The DSE-TTS architecture splits the responsibility of speaker identity between two distinct modules:

  • Acoustic Model (txt2vec): This module is responsible for predicting the linguistic speaking style. During cross-lingual inference, it is fed the speaker embedding of a native speaker of the target language. This ensures the predicted rhythm, stress, and intonation patterns are authentic to the language.
  • Vocoder (vec2wav): This module transforms the neutral VQ features into the final waveform. It is fed the target speaker’s embedding (an X-vector). Its role is to "paint" the target speaker's unique vocal timbre onto the native-style linguistic framework provided by the acoustic model.

3. Pitch Distribution Shifting

Because different speakers have different physical vocal ranges, simply applying a native speaker's prosody to a target speaker's voice can sound unnatural. DSE-TTS implements a pitch shifting mechanism that adjusts the predicted pitch distribution of the native reference to match the mean and standard deviation of the target speaker, ensuring the prosody fits the physical characteristics of the voice.

Experimental Setup

The model was evaluated using a diverse set of datasets:

  • Mandarin (ZH): Aishell3 (60 hours, 142 speakers).
  • English (EN): LibriTTS (74 hours, 228 speakers).
  • Low-Resource Languages: German (DE) and Spanish (ES) from the M-AILABS dataset (6 hours each).
  • Pre-training: The wav2vec 2.0 feature extractor was pre-trained on 10,000 hours of Mandarin data to ensure robust feature extraction.

Results and Discussion

DSE-TTS was compared against SANE-TTS, a state-of-the-art multilingual baseline. The results across several metrics demonstrated the superiority of the dual-embedding approach:

  • Nativeness (NMOS): In cross-lingual scenarios (e.g., synthesizing a Mandarin speaker in English), DSE-TTS outperformed the baseline by over 0.3 points on a 5-point scale, indicating a much more natural, native-sounding accent.
  • Intelligibility: The model showed significantly lower Word Error Rates (WER). For example, in Spanish synthesis, DSE-TTS achieved a WER of 13.5% compared to 17.9% for the baseline.
  • Speaker Similarity: Despite the shift in accent to match the target language, Speaker Embedding Cosine Similarity (SECS) remained high. This confirms that the vocoder successfully applied the target speaker's identity to the native linguistic structure.

Conclusion

DSE-TTS represents a significant step forward in cross-lingual speech synthesis. By identifying VQ features as a superior disentangled representation and employing a dual-embedding strategy, the framework successfully decouples "how" a language is spoken from "who" is speaking it. This allows for the creation of highly native-sounding speech that remains faithful to the original speaker's identity, even across vast linguistic boundaries.

Summary updated

Abstract

Although high-fidelity speech can be obtained for intralingual speech synthesis, cross-lingual text-to-speech (CTTS) is still far from satisfactory as it is difficult to accurately retain the speaker timbres(i.e. speaker similarity) and eliminate the accents from their first language(i.e. nativeness). In this paper, we demonstrated that vector-quantized(VQ) acoustic feature contains less speaker information than mel-spectrogram. Based on this finding, we propose a novel dual speaker embedding TTS (DSE-TTS) framework for CTTS with authentic speaking style. Here, one embedding is fed to the acoustic model to learn the linguistic speaking style, while the other one is integrated into the vocoder to mimic the target speaker's timbre. Experiments show that by combining both embeddings, DSE-TTS significantly outperforms the state-of-the-art SANE-TTS in cross-lingual synthesis, especially in terms of nativeness.
BibTeX
@inproceedings{liu2023dse,
  author = {Sen Liu and Yiwei Guo and Chenpeng Du and Xie Chen and Kai Yu},
  title = {{DSE-TTS: Dual Speaker Embedding for Cross-Lingual Text-to-Speech}},
  booktitle = {Interspeech},
  year = {2023},
  doi = {10.21437/interspeech.2023-363},
}
Figure