Abstract
Voice Conversion (VC) aims to transform the speaker identity of a source utterance while preserving its linguistic content and prosodic nuances. A persistent challenge in VC is the effective disentanglement of speaker identity from other speech components, particularly when dealing with similar-sounding speakers. This paper presents CLN-VC, a text-free framework that leverages contrastive learning and a novel speaker fusion module to enhance speaker discriminability. By generating "hard" synthetic negative samples, CLN-VC forces the model to learn more precise boundaries between speaker identities. Experimental results on the VCTK dataset demonstrate that CLN-VC outperforms existing baselines in both many-to-many and zero-shot conversion tasks, achieving superior similarity and lower acoustic distortion.
1. Introduction
The goal of Voice Conversion (VC) is to modify the non-linguistic information of a speech signal—specifically the speaker's timbre—while maintaining the original linguistic content and prosody. Achieving this requires a high degree of "disentanglement," where the model cleanly separates identity, content, and style. Traditional methods often struggle when the source and target speakers share similar vocal characteristics, such as pitch range or resonance.
While contrastive learning has emerged as a powerful tool for identity representation, its effectiveness is often limited by the quality of negative samples. If a dataset lacks speakers with subtle vocal differences, the model may fail to learn the nuances required for high-fidelity conversion. Furthermore, many state-of-the-art models rely on text transcripts for training, which restricts their applicability to unlabeled, "in-the-wild" audio data. CLN-VC addresses these limitations through a text-free architecture and a specialized negative sample augmentation strategy.
2. Methodology
The CLN-VC framework is built upon three primary components designed to isolate and manipulate speech features without textual supervision.
2.1 VQ-based Content Encoder
To achieve text-free conversion, the model utilizes a Vector Quantization (VQ) based content encoder. This module extracts discrete "acoustic units" from the input speech, which represent linguistic content independently of the speaker's voice. To further ensure that no speaker-specific information leaks into the content features, the authors implement a Gradient Reversal Layer (GRL). This adversarial training component "scrubs" identity information, forcing the content encoder to focus solely on phonetics.
2.2 Fine-Grained Style Modeling
Prosody and timbre are handled through a dual-path reference encoder:
- Global Speaker Embedding (GSE): This captures the overarching identity and timbre of the speaker. It is designed to be utterance-invariant and serves as the primary target for identity transfer.
- Local Prosody Embedding (LPE): This captures the time-varying elements of speech, such as rhythm, stress, and pitch contours. By separating these two, the model can transfer a target speaker's timbre (GSE) while retaining the source speaker's original delivery (LPE).
2.3 Speaker Fusion Module
The core innovation of CLN-VC is the Contrastive Learning with Negative samples augmentation. To improve the model's ability to distinguish between similar voices, the authors introduce a Speaker Fusion Module that creates "hard negative samples" during training. Instead of simply comparing a speaker to a different person in the dataset, the model must distinguish a real speaker from a synthetic "hybrid" speaker.
Two fusion methods were evaluated:
- Linear Fusion: A straightforward mixing of raw audio signals in the time domain.
- Dynamic Fusion: A more sophisticated approach using an attention mechanism within the feature space to create a realistic, hybrid speaker embedding. This method forces the encoder to learn sharper decision boundaries by presenting it with challenging, near-miss identities.
3. Experimental Setup
The model was trained and evaluated using the VCTK Corpus, which contains high-quality recordings from 108 native English speakers with various accents. The training set consisted of 17,262 recordings from 50 speakers. Evaluation was conducted across two scenarios:
- Many-to-Many VC: Converting between speakers seen during the training phase.
- Zero-Shot VC: Converting to speakers that the model never encountered during training, testing the generalizability of the learned embeddings.
4. Results and Discussion
CLN-VC was evaluated using both objective acoustic metrics and subjective human listening tests.
4.1 Similarity and Quality
In many-to-many tasks, CLN-VC achieved a Similarity Mean Opinion Score (SMOS) of 3.79, significantly higher than baseline models like AVQVC (3.18) and SpeechSplit2 (3.35). This suggests that the contrastive learning approach with hard negatives successfully improves identity transfer.
4.2 Zero-Shot Performance
The "Dynamic Fusion" variant proved particularly robust in zero-shot scenarios, maintaining an SMOS of 3.62. While other models often see a sharp decline in performance when encountering unseen speakers, CLN-VC's learned identity space is sufficiently generalized to handle novel voices with high fidelity.
4.3 Acoustic Distortion and Naturalness
Objective measurements using Mel-Cepstral Distortion (MCD) showed that CLN-VC produced outputs with lower distortion (approx. 5.08) compared to baselines (which typically scored above 5.24). Furthermore, when tested against "fake speech" detectors like Resemblyzer, CLN-VC's outputs were frequently identified as real voices, indicating a high level of naturalness and acoustic quality.
5. Conclusion
CLN-VC demonstrates that the quality of voice conversion can be significantly enhanced by focusing on the discriminative power of speaker embeddings. By utilizing a Speaker Fusion Module to generate hard negative samples, the model learns to navigate the subtle nuances of human identity. While the current approach relies on high-quality multi-speaker data for training, future work will explore more complex transformation schemes to further improve zero-shot capabilities in diverse, real-world environments.
Summary updated