TL;DR
NVC-Net is a novel end-to-end adversarial voice conversion system that directly operates on raw audio waveforms, eliminating the need for separate vocoders. This approach achieves significantly faster inference speeds and competitive conversion quality, enabling both many-to-many and zero-shot voice conversion.
What Problem Does This Paper Address?
Traditional voice conversion (VC) methods often rely on intermediate acoustic features and a separate vocoder to reconstruct the final audio. This two-stage process introduces a dependency on the vocoder's quality, which can lead to artifacts and limitations. Furthermore, many existing methods require parallel training data (the same speech content from different speakers), which is difficult and time-consuming to collect. While non-parallel methods using Generative Adversarial Networks (GANs) have emerged, they still face challenges in achieving high-fidelity audio and efficient inference. A significant gap also exists in "zero-shot" VC, where the system needs to convert to a speaker it hasn't encountered during training.
Methods
NVC-Net tackles these challenges with an end-to-end adversarial architecture that directly processes raw audio waveforms. The core idea is to disentangle speaker identity from linguistic content. The system comprises:
- Content Encoder ($E_c$): Extracts speaker-invariant linguistic information from the raw audio.
- Speaker Encoder ($E_s$): Learns a speaker embedding that captures the unique voice characteristics.
- Generator ($G$): Reconstructs the high-fidelity audio waveform conditioned on both the content and speaker embeddings.
- Discriminators ($D^{(k)}$): A set of discriminators operating at different audio resolutions to ensure the generated audio is indistinguishable from real speech.
Key novelties include:
- End-to-End Raw Audio Processing: Eliminates the need for a separate vocoder, avoiding feature mismatch issues and improving efficiency.
- Adversarial Training for Disentanglement: Uses GANs to learn to separate content and speaker information.
- Zero-Shot Capability: Achieved by regularizing the speaker latent space with a Kullback-Leibler (KL) divergence loss, making speaker embeddings robust and allowing for sampling from a prior distribution for unseen speakers.
- Non-Autoregressive and Fully Convolutional: This design contributes to its remarkably fast inference speed.
Datasets
The primary dataset used for training and evaluation is the VCTK dataset [46], which contains utterances from 109 English speakers. For zero-shot experiments, a subset of speakers were designated as "seen" and "unseen" during training.
Key Insights and Ideas
- End-to-end raw audio processing is a viable and efficient path for VC: By directly modeling the waveform, NVC-Net bypasses the limitations and potential artifacts introduced by intermediate representations and vocoders.
- Adversarial learning is powerful for disentanglement: The GAN framework effectively learns to separate speaker identity from linguistic content, a crucial step for flexible voice conversion.
- Latent space regularization is key for zero-shot VC: Applying KL divergence to the speaker embedding space makes it smooth and robust, enabling generalization to unseen speakers without explicit training data for them.
- Speed and quality are not mutually exclusive: NVC-Net demonstrates that achieving high-quality voice conversion doesn't necessarily require slow, autoregressive models.
Key Results
- Competitive Conversion Quality: NVC-Net achieves competitive results in both traditional many-to-many and zero-shot voice conversion settings, comparable to state-of-the-art methods.
- Exceptional Inference Speed: NVC-Net is orders of magnitude faster than other methods, generating audio at over 3600 kHz on a V100 GPU. This makes real-time applications feasible.
- Efficient Model Size: The model is significantly smaller than some competing end-to-end methods like Blow.
- Effective Disentanglement: Speaker identification accuracy is high when using speaker embeddings, and significantly lower when using content embeddings, confirming successful disentanglement.
Limitations and Future Work
- Limited Expressive Speech: The VCTK corpus used for training has limited expressive speech, which might affect the diversity of intonation patterns in synthesized speech.
- Lack of Explicit Controls: The current model lacks explicit controls over other speech aspects like rhythm and prosody.
- Future Directions: The authors suggest extending NVC-Net to other speech conversion tasks such as music and singing voice synthesis. They also highlight the potential for real-time applications on local devices for privacy preservation.
Summary updated