TL;DR
This paper introduces WeStarGAN, a novel training method for non-parallel voice conversion that improves the quality and speaker similarity of synthesized speech. By weighting generated samples based on how well they fool the discriminator, WeStarGAN trains a stronger generator, leading to more robust and higher-fidelity voice transformations.
What Problem Does This Paper Address?
Voice Conversion (VC) aims to change a speaker's voice while preserving the linguistic content. Traditionally, this required parallel speech data (the same sentences spoken by both source and target speakers), which is difficult and expensive to collect. While non-parallel methods have emerged, generating high-quality, natural-sounding speech that accurately captures the target speaker's identity remains a significant challenge. Existing methods like StarGAN-VC offer many-to-many conversion but can still struggle with audio quality and speaker similarity.
Methods
The core innovation in this paper is the Weighted StarGAN (WeStarGAN) approach, which builds upon the StarGAN-VC framework. StarGAN-VC uses Generative Adversarial Networks (GANs) to learn mappings between multiple speaker domains without parallel data. WeStarGAN enhances this by incorporating a novel training algorithm inspired by "Weighted GANs" (WeGAN).
The key idea is to adjust the influence of generated "fake" speech samples during the generator's training. Instead of treating all fake samples equally, WeStarGAN assigns weights:
- Higher weights are given to fake samples that are more convincing and closely resemble real speech, effectively "fooling" the discriminator.
- Lower weights are assigned to fake samples that the discriminator easily identifies as fake.
This weighting is achieved by using the discriminator's output to calculate a weight for each generated sample. This process, rooted in game theory principles, aims to strengthen the generator by focusing its learning on the most challenging and realistic samples. The paper also leverages the stability benefits of Wasserstein GAN with Gradient Penalty (WGAN-GP) within this weighted framework.
The WeStarGAN architecture retains the three main loss functions from StarGAN-VC:
- Adversarial Loss: Standard GAN loss for generating realistic speech.
- Domain Classification Loss: Ensures the converted speech belongs to the target speaker's domain.
- Reconstruction Loss: Preserves the linguistic content of the original speech.
Datasets
The experiments were conducted using the CMU Arctic database. This database contains speech from four speakers (two male, two female). Although the database includes parallel speech, the authors state they randomly selected training data to align with their non-parallel approach.
Key Insights and Ideas
- Weighting for Generator Strength: The core insight is that by strategically weighting fake samples, the generator can be trained to be more robust and produce higher-quality outputs, even when dealing with challenging conversions. This is particularly relevant for improving the generator's ability to mimic subtle vocal characteristics.
- Game Theory in GAN Training: Applying game theory principles to assign weights based on how well a generated sample fools the discriminator provides a principled way to guide the generator's learning process.
- Bridging the Gap in Non-Parallel VC: WeStarGAN offers a practical improvement for non-parallel voice conversion, addressing the persistent challenge of achieving both high speech quality and accurate speaker identity without requiring aligned data.
- Potential for Lighter Models: The authors suggest that a stronger generator trained with this method might enable the use of lighter neural network architectures in the future, which is crucial for deployment on resource-constrained devices.
Key Results
Subjective evaluation experiments, including "ABX" and "AB" listening tests with fifteen native and non-native English listeners, showed significant improvements:
- Speaker Similarity: WeStarGAN achieved a 65% preference score, outperforming the baseline StarGAN-VC.
- Speech Quality (Naturalness): The proposed method also significantly outperformed the baseline in generating good speech quality.
- "No Preference" Option: A notable 17% of preferences for speaker similarity indicated that the baseline and WeStarGAN produced similar speaker characteristics in some cases, suggesting the baseline is already competitive.
Limitations and Future Work
- Computational Cost: While the authors state the additional computational cost is minor, the paper focuses on the generator's weighting. Future work could explore adding weights to the discriminator as well.
- Network Architectures: The paper mentions that lighter generators could be trained using this method. Future research could investigate specific lightweight architectures and their performance with WeStarGAN.
- Further Evaluation: The authors suggest a more extensive study of network architectures and the application of weighting to the discriminator.
Summary updated