TL;DR
This paper introduces a novel cross-lingual voice cloning method that leverages bottleneck (BN) features from speaker-independent ASR systems. By using BN features as a bridge, the approach enables cloning a target speaker's voice into a different language with minimal text-free audio samples, overcoming limitations of previous methods.
What Problem Does This Paper Address?
Voice cloning, the ability to synthesize speech in a specific person's voice, has seen significant advancements. However, extending this capability across different languages (cross-lingual voice cloning) remains a challenge. Existing methods often require bilingual speakers, large amounts of multi-speaker audio-text data, or complex speaker/language encoding modules, making them impractical for real-world applications. This paper aims to simplify and improve cross-lingual voice cloning by requiring only a few text-free audio samples of the target speaker and avoiding the need for bilingual data or complex encoding.
Methods
The proposed framework consists of two main stages: training and synthesizing.
Training Stage: 1. Latent Prosody Model Training: A Tacotron2-based model is trained using audio-text pairs in the target language (e.g., Mandarin). This model learns to predict bottleneck (BN) features from text input, effectively capturing the prosodic and linguistic information. 2. Acoustic Model Training: A separate acoustic model (based on CBHG architecture) is trained using multi-speaker audio data in the target language. This model learns to translate the BN features into acoustic features. 3. Speaker Adaptation: For a new target speaker, the acoustic model is fine-tuned using a small amount of their speech without requiring corresponding text. The BN features from this speech serve as the input for fine-tuning.
Synthesizing Stage: 1. Given a text input in the target language, the trained latent prosody model predicts the corresponding BN features. 2. These BN features are then fed into the fine-tuned acoustic model to generate acoustic features. 3. Finally, a speaker-independent neural vocoder (LPCNet) converts these acoustic features into synthesized speech in the target speaker's voice.
The core innovation lies in using bottleneck (BN) features extracted from a speaker-independent ASR system. These features are hypothesized to be language-independent and capture essential linguistic and prosodic information, acting as a universal representation.
Datasets
- Latent Prosody Model Training: DB-1 corpus (12 hours of Mandarin female data).
- Acoustic Model Training: THCHS-30 (60 speakers, ~250 sentences each, Mandarin).
- Evaluation: The paper uses Mandarin male (MM) and female (MF) speakers for intra-lingual evaluation and English male (EM) and female (EF) speakers for cross-lingual evaluation. For fine-tuning, a few minutes of audio from the target speaker are used. For testing, 30 sentences not used in training are selected.
Key Insights and Ideas
- BN Features as a Universal Bridge: The paper strongly advocates for BN features as a language- and speaker-independent representation that can effectively bridge different languages and speakers. This is a key enabler for cross-lingual cloning.
- Text-Free Adaptation: The ability to fine-tune the acoustic model using only audio samples of the target speaker (without text) significantly simplifies the adaptation process and reduces data requirements.
- Decoupling Prosody and Speaker Identity: The latent prosody model handles the text-to-prosody mapping, while the acoustic model (and its adaptation) focuses on mapping BN features to acoustic features, implicitly learning speaker characteristics. This separation simplifies learning.
- Efficiency and Simplicity: The proposed approach avoids complex speaker/language embeddings and bilingual data, making the system simpler to build and train.
Key Results
- Intra-lingual Performance: The proposed approach outperforms a baseline speaker adaptation method (Tacotron2 fine-tuned with audio-text pairs) in both speech naturalness and speaker similarity for Mandarin speakers, especially with limited data.
- Cross-lingual Performance: The system successfully synthesizes Mandarin speech for English speakers, achieving similarity MOS scores above 3.5, demonstrating good generalization to new cross-lingual speakers.
- Data Efficiency: The method works well with only a few minutes of target speaker audio for adaptation.
Limitations and Future Work
- Lack of Expressiveness: The paper acknowledges that the synthesized speech can lack expressiveness in stress and mood, as BN features might lose some energy information.
- Fixed Prosodic Style: The latent prosody model is trained on a single speaker's corpus, meaning the synthesized speech adopts a fixed prosodic style.
Future work suggested includes:
- Synthesizing more expressive and personalized speech.
- Transferring the prosodic style of the target speaker during voice cloning.
Summary updated