TL;DR
This paper introduces CycleVAE-VC, a novel non-parallel voice conversion method that leverages a cyclic mapping flow to improve spectral modeling. By recycling converted speech features back into the system, it indirectly optimizes the conversion process, leading to more accurate and higher-quality converted speech, especially in challenging cross-gender scenarios.
What Problem Does This Paper Address?
Voice conversion (VC) aims to transform a source speaker's voice into a target speaker's voice while preserving linguistic content. A major challenge in VC is the need for parallel data – recordings of the same speech content from both source and target speakers. Collecting such datasets is often impractical. Non-parallel VC methods, which don't require paired recordings, are thus highly desirable.
Variational Autoencoders (VAEs) have shown promise for non-parallel VC by learning a latent representation of speech. However, conventional VAE-based VC struggles because the conversion process (transforming source to target) cannot be directly optimized when the target speech is not available during training. This leads to suboptimal conversion quality. The paper addresses this limitation by proposing a method to indirectly optimize the conversion flow in a non-parallel setting.
Methods
The core innovation is the CycleVAE-based Voice Conversion (CycleVAE-VC). It builds upon the Variational Autoencoder (VAE) framework but introduces a "cyclic mapping flow." Here's how it works:
- Standard VAE-VC (for context): A VAE encoder maps input speech features to a latent space. A decoder then reconstructs the original speech spectra using the latent features and a "speaker code" representing the source speaker's identity. For conversion, the decoder uses the same latent features but a target speaker code to generate converted spectra. The problem is that the conversion path (source latent + target code -> converted spectra) isn't directly optimized.
- CycleVAE-VC:
- Conversion: The system converts source speech features (X) into target speech spectra ($\hat{s}^{(y|x)}$) using the latent representation and the target speaker code ($c^{(y)}$).
- Recycling: Crucially, these converted target spectra ($\hat{s}^{(y|x)}$) are then treated as input to the encoder again, along with the source speaker code ($c^{(x)}$). This generates "cyclic reconstructed spectra" ($\hat{s}^{(x)}$).
- Indirect Optimization: The loss function now includes terms for reconstructing the original source spectra and for reconstructing the cyclic spectra. By optimizing the reconstruction of these cyclic spectra, the system indirectly forces the conversion process to produce features that are "reversible" or consistent, thereby improving the conversion accuracy.
- Multi-cycle: This cyclic process can be repeated for multiple cycles (N), further refining the conversion.
The paper uses a recurrent neural network (RNN) architecture with dilated convolutional layers and Gated Recurrent Units (GRUs) for the encoder and decoder.
Datasets
The study utilizes a subset of the Voice Conversion Challenge (VCC) 2018 dataset. This dataset includes four speakers: SF1 (Source Female 1), SM1 (Source Male 1), TF1 (Target Female 1), and TM1 (Target Male 1).
- Training: 81 utterances were used. To create a non-parallel setup, the first 40 utterances were assigned to the source speaker and the remaining 41 to the target speaker.
- Testing: 35 utterances were used.
The paper does not introduce a new benchmark dataset.
Key Insights and Ideas
- The "Cycle" is Key: The core insight is that by forcing the system to reconstruct speech from its own converted output, we indirectly guide the conversion process towards better accuracy. This is a clever way to leverage the VAE's reconstruction capabilities for a task (conversion) where direct supervision is missing.
- Latent Space Correlation: The cyclic process appears to encourage the latent representations of different speakers to become more similar (higher cosine similarity), particularly for phonetic attributes. This suggests that CycleVAE-VC learns a more disentangled and robust latent space.
- Cross-Gender Conversion Benefit: The improvements are most pronounced in cross-gender conversions, indicating that the cyclic approach is particularly effective when speaker characteristics are significantly different.
- Reconstruction vs. Conversion Trade-off: The paper notes that while conventional VAEs might achieve good reconstruction accuracy, this doesn't guarantee good conversion. CycleVAE-VC aims to balance both.
Key Results
- Objective Evaluation (Mel-Cepstral Distortion - MCD):
- CycleVAE-VC achieved higher accuracy of converted spectra and lower accuracy of reconstructed spectra compared to conventional VAE.
- Table 1 shows that CycleVAE-based VC, especially when combined with GV-postfiltering, consistently outperformed conventional VAE across different speaker pairs, with MCD values generally lower. For example, for SF1-TF1 conversion, CycleVAE-CV achieved 6.42 dB MCD, compared to 6.95 dB for conventional VAE.
- Latent Feature Correlation (Cosine Similarity):
- Figure 4 demonstrates that CycleVAE-VC generates latent features with higher correlation degree between source and target speakers compared to conventional VAE. This trend holds for both training and testing sets.
- Subjective Evaluation (Perceptual Tests):
- Quality Preference: Listeners preferred CycleVAE-VC's converted speech 59.17% of the time overall, significantly outperforming conventional VAE (40.83%). This preference was especially strong in cross-gender conversions (70.83% for CycleVAE-VC).
- Speaker Similarity Preference: CycleVAE-VC also significantly improved speaker similarity, with listeners preferring its conversions 61.00% of the time overall, and a striking 68.00% in cross-gender conversions.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while reconstruction performance is important, it's not the sole measure of effective disentanglement or conversion.
- The performance of conventional VAE-based VC can be significantly degraded in cross-gender conversions, highlighting a weakness that CycleVAE-VC addresses but doesn't entirely eliminate.
- Future Work:
- Development of many-to-many VC systems.
- Incorporating discrete latent spaces.
- Exploring better prior distributions for the latent space.
- Integrating i-vectors for speaker embedding.
- Using additional classifier networks for improved speaker disentanglement.
- Combining with neural waveform generators (like WaveNet) for more natural-sounding synthesized speech.
Summary updated