EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion

Advait Joglekar, Divyanshu Singh, Rooshil Rohit Bhatia, Srinivasan UmeshEMNLP 2025

TL;DR

This paper introduces EZ-VC, a novel zero-shot voice conversion system that achieves state-of-the-art results with a remarkably simple architecture. By combining pre-trained self-supervised speech representations with a conditional flow matching decoder, EZ-VC enables high-quality, any-to-any voice conversion without needing multiple encoders for feature disentanglement, even across unseen languages.

What Problem Does This Paper Address?

Voice Conversion (VC) aims to transform a speaker's voice while preserving the linguistic content. While recent advancements have improved "zero-shot" VC (converting to/from unseen speakers), a significant challenge remains in achieving this across different languages and accents. Existing methods often struggle with cross-lingual scenarios and may require complex architectures with multiple encoders to disentangle speaker identity, linguistic content, and prosody. Furthermore, many high-quality speech synthesis techniques, like diffusion models, are still under-explored in the context of zero-shot VC. This paper tackles the need for a simpler, more generalizable, and highly effective zero-shot VC system that excels in cross-lingual settings.

Methods

EZ-VC proposes a streamlined architecture that relies on two main components:

  1. Pre-trained Speech Encoder: The system leverages an off-the-shelf, pre-trained self-supervised learning (SSL) encoder, specifically Xeus (trained on a massive multilingual dataset). This encoder extracts frame-level speech representations.
  2. Quantization and K-Means Clustering: The features from the 14th layer of the Xeus encoder are quantized using a k-means clustering model. This process generates discrete speech units that capture linguistic content. The k-means model is trained on a diverse dataset of English and Indian languages.
  3. Conditional Flow Matching (CFM) Speech Decoder: A trainable, non-autoregressive CFM-based speech decoder (specifically, the F5-TTS architecture) is used. This decoder takes the discrete speech units (representing content) and a target speaker's mel-spectrogram (representing speaker identity) as input to generate the converted speech.

A key innovation is the elimination of the need for multiple encoders to disentangle speaker and content features. EZ-VC achieves this by using the discrete units for content and the target speaker's mel-spectrogram for identity, effectively disentangling them within the CFM decoder. The training is entirely self-supervised, requiring no labeled data.

Datasets

The paper utilizes a substantial collection of publicly available datasets for training and evaluation:

  • English: 3060 hours, including LibriSpeech (American accents), Vox Populi (European accents), and NPTEL lectures (Indian English accents).
  • Indian Languages: 9780 hours across Bengali, Hindi, Tamil, Telugu, and Kannada, sourced from Vaani, Common Voice, and datasets from IIIT-H and IIT-M.

The combined dataset comprises 12840 hours of speech across various languages and accents, aiming to promote robust generalization.

Key Insights and Ideas

  • Simplicity is Power: The core insight is that a surprisingly simple architecture, combining a powerful pre-trained SSL encoder with a CFM decoder, can achieve state-of-the-art zero-shot VC. This challenges the notion that complex multi-encoder systems are always necessary.
  • Cross-Lingual Generalization: The use of a multilingual SSL encoder (Xeus) and training on diverse languages is crucial for achieving effective cross-lingual zero-shot VC, even for languages unseen during k-means training.
  • Discrete Units as Content Carriers: Quantized discrete speech units, derived from SSL models, effectively capture linguistic content and prosody, serving as a robust input for speech generation.
  • CFM Decoders Excel: Conditional Flow Matching based decoders demonstrate a strong capability to capture speech styles and generalize well, making them highly suitable for VC tasks.

Key Results

EZ-VC demonstrates superior performance compared to several state-of-the-art baselines across various metrics:

  • Subjective Evaluation (MOS): EZ-VC achieved the highest Naturalness Mean Opinion Score (NMOS) and Similarity Mean Opinion Score (SMOS) among all tested models, indicating highly natural and speaker-similar outputs.
  • Objective Evaluation (SSIM & UTMOS): EZ-VC outperformed baselines like Seed-VC, Vec2wav 2.0, and Diff-HierVC in Speaker Similarity (SSim) and UTMOS scores. For instance, on seen languages (English), EZ-VC achieved an SSIM of 87.3 and UTMOS of 3.76.
  • Cross-Lingual Performance: EZ-VC showed strong performance on unseen languages (German and Spanish), achieving comparable or better SSIM and UTMOS scores than Seed-VC, highlighting its generalization capabilities.

Limitations and Future Work

  • Reliance on Pre-trained Encoder: The performance of EZ-VC is inherently tied to the quality and multilingual capabilities of the pre-trained speech encoder. If the encoder is trained on only one language, the generalization might be limited.
  • Computational Requirements: Despite its architectural simplicity, the paper notes that the computational requirements for EZ-VC might still be comparable to or higher than some previous methods.
  • Potential for Misuse: The paper acknowledges the risk of enabling dangerous deepfakes due to the high quality and cross-lingual capabilities of the model.

Future work could explore further simplifying the architecture, optimizing computational efficiency, and investigating the underlying representations captured by discrete units.

Summary updated

BibTeX
@inproceedings{joglekar2025ez,
  author = {Advait Joglekar and Divyanshu Singh and Rooshil Rohit Bhatia and Srinivasan Umesh},
  title = {{EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure