TL;DR
This paper introduces O_O-VC, a novel voice conversion (VC) framework that leverages synthetic speech data to train models for one-to-one alignment between source and target voices. This approach bypasses the need for complex feature disentanglement and reconstruction, leading to improved intelligibility and speaker similarity, especially in zero-shot scenarios.
What Problem Does This Paper Address?
Traditional voice conversion (VC) methods often struggle to effectively separate linguistic content from speaker identity. This challenge leads to information loss during training and can result in converted speech that retains unwanted speaker characteristics (speaker leakage) or lacks naturalness. Existing approaches often rely on complex feature disentanglement techniques or require parallel datasets, which are not always available. Furthermore, achieving "any-to-any" VC, especially for unseen speakers and languages (zero-shot VC), remains a significant hurdle. This paper aims to address these limitations by proposing a new training paradigm that simplifies the VC process and enhances generalization capabilities.
Methods
The core innovation of O_O-VC lies in its use of synthetic data for training. Instead of relying on real-world audio and complex disentanglement, the authors generate synthetic speech pairs that share identical linguistic content but differ in speaker identity. This is achieved using a high-quality, multi-speaker text-to-speech (TTS) system (specifically VITS).
The training process is divided into two phases:
- Phase 1: Synthetic Data Training: The model is trained on the generated synthetic pairs. This phase focuses on learning independent speaker representations using a pre-trained WavLM model (which is frozen) and a content extractor. The goal is to establish a direct, one-to-one mapping between source and target voices.
- Phase 2: Fine-Tuning Adaptation: The model is further fine-tuned on a large, real-world speech corpus (LibriSpeech). This phase uses a reconstruction-based objective and helps the model adapt to new speakers and domains without requiring transcriptions. Crucially, the WavLM and content extractor remain frozen to preserve the speaker-independent representations learned in Phase 1.
A key technical detail is the Fundamental Frequency (F0) Adjustment. To address potential F0 mismatches between source and target audio, the F0 of the target audio is extracted and used as an additional conditioning feature during decoding, improving pitch consistency.
Datasets
- Training (Phase 1): Synthetic speech generated by a publicly available pretrained VITS model, mimicking the VCTK training set.
- Training (Phase 2): LibriSpeech dataset (train-clean-360 and train-clean-100 subsets), totaling approximately 460 hours of speech from 1,172 speakers.
- Evaluation: LibriSpeech test-clean subset for zero-shot VC. For adaptation to new languages, the authors used:
- Chinese (ZH): AISHELL-3 dataset.
- Italian (IT): Multilingual LibriSpeech (MLS) training subset.
- Vietnamese (VI): A clean test set constructed from the VIVOS dataset.
Key Insights and Ideas
- Synthetic Data as a Game-Changer: The paper demonstrates that high-quality synthetic data can serve as a powerful and direct supervisory signal for VC, simplifying the training process and eliminating the need for complex feature disentanglement.
- One-to-One Alignment is Key: By using perfectly aligned synthetic pairs, the model can learn a direct mapping, which is more effective for VC than indirect reconstruction-based methods.
- Two-Phase Training for Robustness: The combination of synthetic data training followed by real-data fine-tuning proves crucial for achieving both good initial representations and robust generalization to unseen speakers and domains.
- F0 Conditioning is Essential for Naturalness: Explicitly conditioning on the target F0 significantly improves the naturalness and pitch accuracy of the converted speech.
- Improved Speaker Independence: The proposed method shows a marked improvement in removing speaker-specific information compared to existing methods, as evidenced by clustering metrics and t-SNE visualizations.
Key Results
- Objective Evaluation: O_O-VC achieved the lowest Word Error Rate (WER) (1.74) and Character Error Rate (CER) (0.53) among the evaluated models, indicating superior intelligibility. It also achieved the second-highest Speaker Encoder Cosine Similarity (SECS) (86.70).
- Subjective Evaluation: The model achieved a good balance between naturalness (MOS: 3.42) and speaker similarity (SMOS: 3.48), outperforming several state-of-the-art methods in zero-shot scenarios.
- Language Adaptation: The model demonstrated effective adaptation to unseen languages (Chinese, Italian, Vietnamese) with significant improvements in CER/WER after phase 2 fine-tuning.
- Speaker Information Removal: Quantitative metrics (ARI, NMI, Silhouette Score) showed that O_O-VC was more effective at removing speaker-specific information compared to FreeVC.
Limitations and Future Work
- TTS System Dependency: The effectiveness of the proposed method is tied to the quality of the TTS system used for synthetic data generation. The authors acknowledge that the influence of different TTS systems on VC performance is not fully explored.
- Future Work: The authors plan to investigate alternative TTS models to gain a deeper understanding of their impact. They also aim to explore the impact of synthetic data generation on VC performance across different TTS systems. Additionally, they suggest investigating alternative methods for achieving more comprehensive understanding of their impact on overall system performance.
Summary updated