Introduction
Cross-lingual voice conversion (XVC) aims to transform a speaker's voice into a target language they do not speak while maintaining their unique vocal identity. While recent advancements have improved speaker similarity, a persistent challenge remains: the "foreign accent" problem. Converted speech often retains the phonetic and prosodic habits of the source language, leading to reduced intelligibility and unnatural-sounding output.
The Intelligibility Gap in XVC
The researchers first established the severity of the "intelligibility gap" in current XVC systems. By analyzing performance from the Voice Conversion Challenge 2020, they found that Word Error Rates (WER) for cross-lingual tasks are significantly higher than for intra-lingual ones. This issue is particularly acute for tonal languages like Mandarin Chinese, where subtle mispronunciations can fundamentally alter word meanings.
Methodology: BNF-to-Waveform with Linguistic Losses
To address these issues, the authors proposed a framework that moves beyond traditional acoustic-only modeling. Their system utilizes Bottleneck Features (BNFs)—speaker-independent phonetic representations—to generate waveforms directly via a modified Parallel WaveGAN (PWG).
The primary innovation is the introduction of two linguistic loss functions during the training phase to supervise the "correctness" of the generated speech:
- Frame-Wise BNF Loss ($L_{BNF}$): This loss ensures that the fine-grained phonetic content of the generated audio matches the input speech at the frame level.
- Segmental ASR Loss ($L_{ASR}$): By passing generated audio through a pre-trained Automatic Speech Recognition (ASR) model, the system calculates the error relative to ground-truth text. This ensures word-level intelligibility.
The model also incorporates language-specific output layers to better handle the distinct acoustic characteristics of English and Mandarin.
Experimental Results
The proposed system, BNF2Wav-BNF-ASR, was evaluated using the EMIME database, which features bilingual speakers for ground-truth comparison. The results demonstrated significant improvements across all metrics:
- Intelligibility: The average WER/CER was slashed from 19.75% in baseline models to 10.18%.
- Acoustic Quality: Mel-Cepstral Distortion (MCD) improved from 8.89 dB to 8.13 dB.
- Subjective Preference: In MUSHRA listening tests, participants rated the proposed model as sounding more "native" and natural than previous state-of-the-art systems.
Crucially, these linguistic corrections did not compromise speaker similarity; the model successfully preserved the target speaker's identity while removing their native accent.
Conclusion and Future Work
By integrating linguistic supervision into the voice conversion pipeline, this research successfully mitigates the foreign accent problem in XVC. While the current model relies on ground-truth text for the ASR loss, future work will explore self-supervised learning techniques to remove this dependency. This would allow the system to be trained on vast amounts of unlabelled data, further enhancing the accessibility and quality of cross-lingual voice conversion.
Summary updated