TL;DR
This paper introduces a novel semi-supervised approach to voice conversion (VC) that effectively leverages both parallel (paired source-target speech) and non-parallel (unpaired source and target speech) data during training. By combining the strengths of traditional statistical VC methods and modern deep learning techniques, their method significantly improves conversion quality, especially when limited parallel data is available, and shows promise for more flexible and cost-effective VC system development.
What Problem Does This Paper Address?
Voice conversion (VC) aims to transform speech from a source speaker into the voice of a target speaker. Traditional methods, like Gaussian Mixture Models (GMMs), often require large amounts of parallel data (the exact same utterance spoken by both source and target speakers) for training. Collecting such parallel data is expensive and time-consuming, making it a significant bottleneck for practical VC applications. While non-parallel data is easier to obtain, using it effectively without sacrificing speech quality or phonetic content is challenging. More recent deep learning approaches have shown success but often demand even larger datasets. This paper addresses the gap by proposing a method that can learn from both parallel and non-parallel data simultaneously, thereby reducing the dependency on scarce parallel data and improving overall system performance and practicality.
Methods
The core of the proposed method is a semi-supervised learning framework built upon a deep generative model, specifically a variational autoencoder (VAE). They frame voice conversion as inferring a shared latent variable sequence ($Z$) that generates both the source ($X$) and target ($Y$) speech features. The key innovation lies in how they handle the training data:
- Amortized Variational Inference: To make the inference process tractable for complex deep models, they employ amortized variational inference. This involves learning an approximate posterior distribution ($q(Z)$) that approximates the true posterior ($p(Z|X)$ or $p(Z|Y)$).
- Shared Latent Space: The model assumes a single latent sequence $Z$ that captures the underlying phonetic content, which is then transformed into speaker-specific features for both source and target. This is inspired by GMM-based VC.
- Hybrid Training Objective: The training objective is a combination of terms that leverage different data types:
- A term for parallel data ($L_{X,Y}$), similar to supervised VAEs.
- A term for non-parallel source data ($L_X$), where only source speech is available.
-
A term for non-parallel target data ($L_Y$), where only target speech is available. This allows the model to learn from all available data, even if it's not perfectly paired.
-
Neural Network Architecture: They adapt a successful deep bidirectional LSTM (DBLSTM) model for VC. The first two layers act as an encoder to infer the latent variable distribution $q(Z)$, and the last two layers act as decoders to generate source and target features from $Z$. This reinterpretation of the DBLSTM within a VAE framework is termed DBLSTM+VAE.
Datasets
The experiments were conducted using the CMU Arctic corpus, specifically the CLB SLT (both females) pair.
- Parallel Data: Generated by pairing utterances from the A and B partitions of the corpus and time-aligning target features to source features using Dynamic Time Warping (DTW).
- Non-parallel Data: Created by using remaining unselected samples from the A and B partitions, ensuring no prompt overlap between source and target non-parallel sets.
- Validation/Testing: A small portion of the A partition was used for validation, and the remaining B partition utterances were used for testing.
The paper explores scenarios with varying numbers of parallel utterances (from 1 to 1000) out of a total budget of 1000 utterances, with non-parallel utterances making up the rest.
Key Insights and Ideas
- The Power of Semi-Supervision for Data Scarcity: The most striking insight is how effectively semi-supervised training bridges the performance gap when parallel data is scarce. The proposed method significantly outperforms purely supervised training in these low-resource scenarios.
- Non-Parallel Data is a Valuable Resource: The study demonstrates that increasing the amount of non-parallel data consistently improves VC performance, even when the amount of parallel data is fixed. This highlights the potential to build robust VC systems with more readily available data.
- Bridging Generative Models and Deep Networks: The paper successfully combines the probabilistic reasoning of generative models (like GMMs) with the representational power of deep neural networks (like LSTMs) by using amortized variational inference. This offers a principled way to leverage both types of data.
- Smooth Interpolation of Performance: The semi-supervised approach smoothly interpolates between purely supervised and unsupervised scenarios, offering a flexible trade-off between data availability and performance.
Key Results
- Improved Quality with Limited Parallel Data: When trained with only 1 or 10 parallel utterances (out of 1000 total), the semi-supervised method achieved significantly better Mean Opinion Scores (MOS) and lower Mel-Cepstral Distortion (MCD) compared to a purely supervised DBLSTM model. For 1 parallel utterance, the MOS was 2.93 vs. 1.64 for the supervised model.
- Comparable Performance with Ample Parallel Data: When a large amount of parallel data (1000 utterances) was available, the semi-supervised method achieved performance comparable to or slightly better than the fully supervised approach.
- Continued Improvement with Non-Parallel Data: Even with just one parallel utterance, increasing the amount of non-parallel data led to improved VC performance, though with diminishing returns.
- Higher Perceptual Quality: Subjective evaluations (MOS) indicated that the semi-supervised training resulted in audio of equal or higher perceptual quality than supervised systems trained with more parallel data.
Limitations and Future Work
- Trade-off Between Parallel and Non-Parallel Data: The authors acknowledge that a trade-off exists in dataset creation between gathering hard-to-obtain parallel data and easier-to-obtain non-parallel data.
- Improving Non-Parallel Data Efficiency: They suggest that future work could focus on devising methods to improve the efficiency of using non-parallel training data, which would further reduce the cost and difficulty of creating VC datasets.
- Exploring Different Architectures: While they adapted a DBLSTM, exploring other deep learning architectures within their semi-supervised framework could yield further improvements.
- Investigating Prompt-Specific Non-Parallel Data: The current non-parallel data setup ensures no prompt overlap. Future work could explore how to best utilize non-parallel data that does contain overlapping prompts.
Summary updated