TL;DR
This paper introduces EZ-VC, a novel zero-shot voice conversion system that achieves state-of-the-art results with a remarkably simple architecture. By combining pre-trained self-supervised speech representations with a conditional flow matching decoder, EZ-VC enables high-quality, any-to-any voice conversion without needing multiple encoders for feature disentanglement, even across unseen languages.
What Problem Does This Paper Address?
Voice Conversion (VC) aims to transform a speaker's voice while preserving the linguistic content. While recent advancements have improved "zero-shot" VC (converting to/from unseen speakers), a significant challenge remains in achieving this across different languages and accents. Existing methods often struggle with cross-lingual scenarios and may require complex architectures with multiple encoders to disentangle speaker identity, linguistic content, and prosody. Furthermore, many high-quality speech synthesis techniques, like diffusion models, are still under-explored in the context of zero-shot VC. This paper tackles the need for a simpler, more generalizable, and highly effective zero-shot VC system that excels in cross-lingual settings.
Methods
EZ-VC proposes a streamlined architecture that relies on two main components:
- Pre-trained Speech Encoder: The system leverages an off-the-shelf, pre-trained self-supervised learning (SSL) encoder, specifically Xeus (trained on a massive multilingual dataset). This encoder extracts frame-level speech representations.
- Quantization and K-Means Clustering: The features from the 14th layer of the Xeus encoder are quantized using a k-means clustering model. This process generates discrete speech units that capture linguistic content. The k-means model is trained on a diverse dataset of English and Indian languages.
- Conditional Flow Matching (CFM) Speech Decoder: A trainable, non-autoregressive CFM-based speech decoder (specifically, the F5-TTS architecture) is used. This decoder takes the discrete speech units (representing content) and a target speaker's mel-spectrogram (representing speaker identity) as input to generate the converted speech.
A key innovation is the elimination of the need for multiple encoders to disentangle speaker and content features. EZ-VC achieves this by using the discrete units for content and the target speaker's mel-spectrogram for identity, effectively disentangling them within the CFM decoder. The training is entirely self-supervised, requiring no labeled data.
Datasets
The paper utilizes a substantial collection of publicly available datasets for training and evaluation:
- English: 3060 hours, including LibriSpeech (American accents), Vox Populi (European accents), and NPTEL lectures (Indian English accents).
- Indian Languages: 9780 hours across Bengali, Hindi, Tamil, Telugu, and Kannada, sourced from Vaani, Common Voice, and datasets from IIIT-H and IIT-M.
The combined dataset comprises 12840 hours of speech across various languages and accents, aiming to promote robust generalization.
Key Insights and Ideas
- Simplicity is Power: The core insight is that a surprisingly simple architecture, combining a powerful pre-trained SSL encoder with a CFM decoder, can achieve state-of-the-art zero-shot VC. This challenges the notion that complex multi-encoder systems are always necessary.
- Cross-Lingual Generalization: The use of a multilingual SSL encoder (Xeus) and training on diverse languages is crucial for achieving effective cross-lingual zero-shot VC, even for languages unseen during k-means training.
- Discrete Units as Content Carriers: Quantized discrete speech units, derived from SSL models, effectively capture linguistic content and prosody, serving as a robust input for speech generation.
- CFM Decoders Excel: Conditional Flow Matching based decoders demonstrate a strong capability to capture speech styles and generalize well, making them highly suitable for VC tasks.
Key Results
EZ-VC demonstrates superior performance compared to several state-of-the-art baselines across various metrics:
- Subjective Evaluation (MOS): EZ-VC achieved the highest Naturalness Mean Opinion Score (NMOS) and Similarity Mean Opinion Score (SMOS) among all tested models, indicating highly natural and speaker-similar outputs.
- Objective Evaluation (SSIM & UTMOS): EZ-VC outperformed baselines like Seed-VC, Vec2wav 2.0, and Diff-HierVC in Speaker Similarity (SSim) and UTMOS scores. For instance, on seen languages (English), EZ-VC achieved an SSIM of 87.3 and UTMOS of 3.76.
- Cross-Lingual Performance: EZ-VC showed strong performance on unseen languages (German and Spanish), achieving comparable or better SSIM and UTMOS scores than Seed-VC, highlighting its generalization capabilities.
Limitations and Future Work
- Reliance on Pre-trained Encoder: The performance of EZ-VC is inherently tied to the quality and multilingual capabilities of the pre-trained speech encoder. If the encoder is trained on only one language, the generalization might be limited.
- Computational Requirements: Despite its architectural simplicity, the paper notes that the computational requirements for EZ-VC might still be comparable to or higher than some previous methods.
- Potential for Misuse: The paper acknowledges the risk of enabling dangerous deepfakes due to the high quality and cross-lingual capabilities of the model.
Future work could explore further simplifying the architecture, optimizing computational efficiency, and investigating the underlying representations captured by discrete units.
Summary updated