a novel cross-lingual voice cloning approach with a few text-free samples

Xinyong Zhou, Hao Che, Xiaorui Wang, Lei XiearXiv 2019 · OpenReview

TL;DR

This paper introduces a novel cross-lingual voice cloning method that leverages bottleneck (BN) features from speaker-independent ASR systems. By using BN features as a bridge, the approach enables cloning a target speaker's voice into a different language with minimal text-free audio samples, overcoming limitations of previous methods.

What Problem Does This Paper Address?

Voice cloning, the ability to synthesize speech in a specific person's voice, has seen significant advancements. However, extending this capability across different languages (cross-lingual voice cloning) remains a challenge. Existing methods often require bilingual speakers, large amounts of multi-speaker audio-text data, or complex speaker/language encoding modules, making them impractical for real-world applications. This paper aims to simplify and improve cross-lingual voice cloning by requiring only a few text-free audio samples of the target speaker and avoiding the need for bilingual data or complex encoding.

Methods

The proposed framework consists of two main stages: training and synthesizing.

Training Stage: 1. Latent Prosody Model Training: A Tacotron2-based model is trained using audio-text pairs in the target language (e.g., Mandarin). This model learns to predict bottleneck (BN) features from text input, effectively capturing the prosodic and linguistic information. 2. Acoustic Model Training: A separate acoustic model (based on CBHG architecture) is trained using multi-speaker audio data in the target language. This model learns to translate the BN features into acoustic features. 3. Speaker Adaptation: For a new target speaker, the acoustic model is fine-tuned using a small amount of their speech without requiring corresponding text. The BN features from this speech serve as the input for fine-tuning.

Synthesizing Stage: 1. Given a text input in the target language, the trained latent prosody model predicts the corresponding BN features. 2. These BN features are then fed into the fine-tuned acoustic model to generate acoustic features. 3. Finally, a speaker-independent neural vocoder (LPCNet) converts these acoustic features into synthesized speech in the target speaker's voice.

The core innovation lies in using bottleneck (BN) features extracted from a speaker-independent ASR system. These features are hypothesized to be language-independent and capture essential linguistic and prosodic information, acting as a universal representation.

Datasets

  • Latent Prosody Model Training: DB-1 corpus (12 hours of Mandarin female data).
  • Acoustic Model Training: THCHS-30 (60 speakers, ~250 sentences each, Mandarin).
  • Evaluation: The paper uses Mandarin male (MM) and female (MF) speakers for intra-lingual evaluation and English male (EM) and female (EF) speakers for cross-lingual evaluation. For fine-tuning, a few minutes of audio from the target speaker are used. For testing, 30 sentences not used in training are selected.

Key Insights and Ideas

  • BN Features as a Universal Bridge: The paper strongly advocates for BN features as a language- and speaker-independent representation that can effectively bridge different languages and speakers. This is a key enabler for cross-lingual cloning.
  • Text-Free Adaptation: The ability to fine-tune the acoustic model using only audio samples of the target speaker (without text) significantly simplifies the adaptation process and reduces data requirements.
  • Decoupling Prosody and Speaker Identity: The latent prosody model handles the text-to-prosody mapping, while the acoustic model (and its adaptation) focuses on mapping BN features to acoustic features, implicitly learning speaker characteristics. This separation simplifies learning.
  • Efficiency and Simplicity: The proposed approach avoids complex speaker/language embeddings and bilingual data, making the system simpler to build and train.

Key Results

  • Intra-lingual Performance: The proposed approach outperforms a baseline speaker adaptation method (Tacotron2 fine-tuned with audio-text pairs) in both speech naturalness and speaker similarity for Mandarin speakers, especially with limited data.
  • Cross-lingual Performance: The system successfully synthesizes Mandarin speech for English speakers, achieving similarity MOS scores above 3.5, demonstrating good generalization to new cross-lingual speakers.
  • Data Efficiency: The method works well with only a few minutes of target speaker audio for adaptation.

Limitations and Future Work

  • Lack of Expressiveness: The paper acknowledges that the synthesized speech can lack expressiveness in stress and mood, as BN features might lose some energy information.
  • Fixed Prosodic Style: The latent prosody model is trained on a single speaker's corpus, meaning the synthesized speech adopts a fixed prosodic style.

Future work suggested includes:

  • Synthesizing more expressive and personalized speech.
  • Transferring the prosodic style of the target speaker during voice cloning.

Summary updated

Abstract

In this paper, we present a cross-lingual voice cloning approach. BN features obtained by SI-ASR model are used as a bridge across speakers and language boundaries. The relationships between text and BN features are modeled by the latent prosody model. The acoustic model learns the translation from BN features to acoustic features. The acoustic model is fine-tuned with a few samples of the target speaker to realize voice cloning. This system can generate speech of arbitrary utterance of target language in cross-lingual speakers' voice. We verify that with small amount of audio data, our proposed approach can well handle cross-lingual tasks. And in intra-lingual tasks, our proposed approach also performs better than baseline approach in naturalness and similarity.
BibTeX
@misc{zhou2019novel,
  author = {Xinyong Zhou and Hao Che and Xiaorui Wang and Lei Xie},
  title = {{a novel cross-lingual voice cloning approach with a few text-free samples}},
  year = {2019},
  eprint = {1910.13276},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.13276},
}
Figure