Optimization of Cross-Lingual Voice Conversion With Linguistics Losses to Reduce Foreign Accents

Yi Zhou, Zhizheng Wu, Xiaohai Tian, Haizhou Li

Introduction

Cross-lingual voice conversion (XVC) aims to transform a speaker's voice into a target language they do not speak while maintaining their unique vocal identity. While recent advancements have improved speaker similarity, a persistent challenge remains: the "foreign accent" problem. Converted speech often retains the phonetic and prosodic habits of the source language, leading to reduced intelligibility and unnatural-sounding output.

The Intelligibility Gap in XVC

The researchers first established the severity of the "intelligibility gap" in current XVC systems. By analyzing performance from the Voice Conversion Challenge 2020, they found that Word Error Rates (WER) for cross-lingual tasks are significantly higher than for intra-lingual ones. This issue is particularly acute for tonal languages like Mandarin Chinese, where subtle mispronunciations can fundamentally alter word meanings.

Comparison of error rates between intra-lingual and cross-lingual voice conversion tasks

Methodology: BNF-to-Waveform with Linguistic Losses

To address these issues, the authors proposed a framework that moves beyond traditional acoustic-only modeling. Their system utilizes Bottleneck Features (BNFs)—speaker-independent phonetic representations—to generate waveforms directly via a modified Parallel WaveGAN (PWG).

The primary innovation is the introduction of two linguistic loss functions during the training phase to supervise the "correctness" of the generated speech:

  1. Frame-Wise BNF Loss ($L_{BNF}$): This loss ensures that the fine-grained phonetic content of the generated audio matches the input speech at the frame level.
  2. Segmental ASR Loss ($L_{ASR}$): By passing generated audio through a pre-trained Automatic Speech Recognition (ASR) model, the system calculates the error relative to ground-truth text. This ensures word-level intelligibility.
The proposed training framework utilizing BNF and ASR losses to ensure linguistic correctness

The model also incorporates language-specific output layers to better handle the distinct acoustic characteristics of English and Mandarin.

Experimental Results

The proposed system, BNF2Wav-BNF-ASR, was evaluated using the EMIME database, which features bilingual speakers for ground-truth comparison. The results demonstrated significant improvements across all metrics:

  • Intelligibility: The average WER/CER was slashed from 19.75% in baseline models to 10.18%.
  • Acoustic Quality: Mel-Cepstral Distortion (MCD) improved from 8.89 dB to 8.13 dB.
  • Subjective Preference: In MUSHRA listening tests, participants rated the proposed model as sounding more "native" and natural than previous state-of-the-art systems.

Crucially, these linguistic corrections did not compromise speaker similarity; the model successfully preserved the target speaker's identity while removing their native accent.

Conclusion and Future Work

By integrating linguistic supervision into the voice conversion pipeline, this research successfully mitigates the foreign accent problem in XVC. While the current model relies on ground-truth text for the ASR loss, future work will explore self-supervised learning techniques to remove this dependency. This would allow the system to be trained on vast amounts of unlabelled data, further enhancing the accessibility and quality of cross-lingual voice conversion.

Summary updated

Abstract

Cross-lingual voice conversion (XVC) transforms the speaker identity of a source speaker to that of a target speaker who speaks a different language. Due to the intrinsic differences between languages, the converted speech may carry an unwanted foreign accent. In this paper, we first investigate the intelligibility of the converted speech and confirm the performance degradation caused by the accent/intelligibility issue. With the goal of generating native-sounding speech, this paper further proposes a novel training scheme with two additional linguistic losses for speech waveform generation: 1) a frame-wise phonetic content loss derived from bottleneck features, and 2) an automatic speech recognition loss on characters. Experiments were conducted between English and Mandarin Chinese conversions. The experimental results confirmed that the generated speech sounds more natural with the proposed linguistic losses and the proposed solution significantly improves speech intelligibility.
BibTeX
@article{zhou2023optimization,
  author = {Yi Zhou and Zhizheng Wu and Xiaohai Tian and Haizhou Li},
  title = {{Optimization of Cross-Lingual Voice Conversion With Linguistics Losses to Reduce Foreign Accents}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2023},
  doi = {10.1109/taslp.2023.3271107},
}
Figure