CLN-VC: Text-Free Voice Conversion Based on Fine-Grained Style Control and Contrastive Learning with Negative Samples Augmentation

Yimin Deng, Xulong Zhang, Jianzong Wang, Ning Cheng, Jing Xiao

Abstract

Voice Conversion (VC) aims to transform the speaker identity of a source utterance while preserving its linguistic content and prosodic nuances. A persistent challenge in VC is the effective disentanglement of speaker identity from other speech components, particularly when dealing with similar-sounding speakers. This paper presents CLN-VC, a text-free framework that leverages contrastive learning and a novel speaker fusion module to enhance speaker discriminability. By generating "hard" synthetic negative samples, CLN-VC forces the model to learn more precise boundaries between speaker identities. Experimental results on the VCTK dataset demonstrate that CLN-VC outperforms existing baselines in both many-to-many and zero-shot conversion tasks, achieving superior similarity and lower acoustic distortion.

1. Introduction

The goal of Voice Conversion (VC) is to modify the non-linguistic information of a speech signal—specifically the speaker's timbre—while maintaining the original linguistic content and prosody. Achieving this requires a high degree of "disentanglement," where the model cleanly separates identity, content, and style. Traditional methods often struggle when the source and target speakers share similar vocal characteristics, such as pitch range or resonance.

While contrastive learning has emerged as a powerful tool for identity representation, its effectiveness is often limited by the quality of negative samples. If a dataset lacks speakers with subtle vocal differences, the model may fail to learn the nuances required for high-fidelity conversion. Furthermore, many state-of-the-art models rely on text transcripts for training, which restricts their applicability to unlabeled, "in-the-wild" audio data. CLN-VC addresses these limitations through a text-free architecture and a specialized negative sample augmentation strategy.

2. Methodology

The CLN-VC framework is built upon three primary components designed to isolate and manipulate speech features without textual supervision.

2.1 VQ-based Content Encoder

To achieve text-free conversion, the model utilizes a Vector Quantization (VQ) based content encoder. This module extracts discrete "acoustic units" from the input speech, which represent linguistic content independently of the speaker's voice. To further ensure that no speaker-specific information leaks into the content features, the authors implement a Gradient Reversal Layer (GRL). This adversarial training component "scrubs" identity information, forcing the content encoder to focus solely on phonetics.

2.2 Fine-Grained Style Modeling

Prosody and timbre are handled through a dual-path reference encoder:

  • Global Speaker Embedding (GSE): This captures the overarching identity and timbre of the speaker. It is designed to be utterance-invariant and serves as the primary target for identity transfer.
  • Local Prosody Embedding (LPE): This captures the time-varying elements of speech, such as rhythm, stress, and pitch contours. By separating these two, the model can transfer a target speaker's timbre (GSE) while retaining the source speaker's original delivery (LPE).

2.3 Speaker Fusion Module

The core innovation of CLN-VC is the Contrastive Learning with Negative samples augmentation. To improve the model's ability to distinguish between similar voices, the authors introduce a Speaker Fusion Module that creates "hard negative samples" during training. Instead of simply comparing a speaker to a different person in the dataset, the model must distinguish a real speaker from a synthetic "hybrid" speaker.

Two fusion methods were evaluated:

  1. Linear Fusion: A straightforward mixing of raw audio signals in the time domain.
  2. Dynamic Fusion: A more sophisticated approach using an attention mechanism within the feature space to create a realistic, hybrid speaker embedding. This method forces the encoder to learn sharper decision boundaries by presenting it with challenging, near-miss identities.

3. Experimental Setup

The model was trained and evaluated using the VCTK Corpus, which contains high-quality recordings from 108 native English speakers with various accents. The training set consisted of 17,262 recordings from 50 speakers. Evaluation was conducted across two scenarios:

  • Many-to-Many VC: Converting between speakers seen during the training phase.
  • Zero-Shot VC: Converting to speakers that the model never encountered during training, testing the generalizability of the learned embeddings.

4. Results and Discussion

CLN-VC was evaluated using both objective acoustic metrics and subjective human listening tests.

4.1 Similarity and Quality

In many-to-many tasks, CLN-VC achieved a Similarity Mean Opinion Score (SMOS) of 3.79, significantly higher than baseline models like AVQVC (3.18) and SpeechSplit2 (3.35). This suggests that the contrastive learning approach with hard negatives successfully improves identity transfer.

4.2 Zero-Shot Performance

The "Dynamic Fusion" variant proved particularly robust in zero-shot scenarios, maintaining an SMOS of 3.62. While other models often see a sharp decline in performance when encountering unseen speakers, CLN-VC's learned identity space is sufficiently generalized to handle novel voices with high fidelity.

4.3 Acoustic Distortion and Naturalness

Objective measurements using Mel-Cepstral Distortion (MCD) showed that CLN-VC produced outputs with lower distortion (approx. 5.08) compared to baselines (which typically scored above 5.24). Furthermore, when tested against "fake speech" detectors like Resemblyzer, CLN-VC's outputs were frequently identified as real voices, indicating a high level of naturalness and acoustic quality.

5. Conclusion

CLN-VC demonstrates that the quality of voice conversion can be significantly enhanced by focusing on the discriminative power of speaker embeddings. By utilizing a Speaker Fusion Module to generate hard negative samples, the model learns to navigate the subtle nuances of human identity. While the current approach relies on high-quality multi-speaker data for training, future work will explore more complex transformation schemes to further improve zero-shot capabilities in diverse, real-world environments.

Summary updated

Abstract

Better disentanglement of speech representation is essential to improve the quality of voice conversion. Recently contrastive learning is applied to voice conversion successfully based on speaker labels. However, the performance of model will reduce in conversion between similar speakers. Hence, we propose an augmented negative sample selection to address the issue. Specifically, we create hard negative samples based on the proposed speaker fusion module to improve learning ability of speaker encoder. Furthermore, considering the fine-grain modeling of speaker style, we employ a reference encoder to extract fine-grained style and conduct the augmented contrastive learning on global style. The experimental results show that the proposed method outperforms previous work in voice conversion tasks.
BibTeX
@inproceedings{deng2023cln,
  author = {Yimin Deng and Xulong Zhang and Jianzong Wang and Ning Cheng and Jing Xiao},
  title = {{CLN-VC: Text-Free Voice Conversion Based on Fine-Grained Style Control and Contrastive Learning with Negative Samples Augmentation}},
  booktitle = {ISPA/BDCloud/SocialCom/SustainCom},
  year = {2023},
  url = {https://openreview.net/forum?id=gcOQ2J6R0r},
}
Figure