Learning Expressive Disentangled Speech Representations with Soft Speech Units and Adversarial Style Augmentation

Yimin Deng, Jianzong Wang, Xulong Zhang, Ning Cheng, Jing Xiao

Introduction

Voice Conversion (VC) is the process of modifying the speaker identity (timbre) of a source utterance while preserving its linguistic content and prosodic nuances. Despite recent advancements in self-supervised learning (SSL), achieving high-fidelity conversion remains a challenge due to the difficulty of perfectly disentangling speech components. This paper introduces SAVC (Soft speech units and Adversarial style augmentation for Voice Conversion), a framework designed to produce expressive, high-quality speech by effectively separating content, timbre, and prosody.

The Problem: Timbre Leakage and Prosody Neglect

Modern VC systems often rely on SSL models like HuBERT to extract content representations. However, these systems frequently encounter two primary obstacles:

  1. Timbre Leakage: SSL representations are not purely linguistic; they often retain residual information about the source speaker's identity. This leads to "leakage," where the converted audio sounds like a hybrid of the source and target speakers, undermining the conversion's authenticity.
  2. Prosody Neglect: Many models prioritize content and timbre at the expense of prosody—the rhythm, pitch, and energy that convey emotion and naturalness. This often results in speech that sounds robotic or fails to capture the expressive intent of the original speaker.

While some researchers use manual signal processing or information bottlenecks to address these issues, these methods can inadvertently degrade overall speech quality.

The SAVC Framework

SAVC addresses these challenges through three core innovations: Soft Speech Units, Adversarial Style Augmentation (ASA), and Prosody Distillation.

Soft Speech Units

Rather than using discrete, quantized speech tokens which can discard subtle linguistic details, SAVC utilizes "soft" units derived from HuBERT-Soft. These units provide a continuous representation that retains more nuance than discrete tokens while remaining significantly more speaker-independent than raw mel-spectrograms.

Adversarial Style Augmentation (ASA)

The central innovation of SAVC is the Adversarial Style Augmentation (ASA) technique. During training, the model applies dynamic statistical perturbations to the mean and standard deviation of speech features. A Gradient Reversal Layer (GRL) is then employed to force the attribute encoder to become invariant to these speaker-specific "style" statistics. This process effectively "scrubs" the source speaker's identity from the content and prosody embeddings, ensuring a cleaner separation of attributes.

Prosody Distillation

To capture expressiveness without requiring explicit pitch or energy labels during inference, SAVC employs a teacher-student distillation setup. A teacher model, trained with explicit access to pitch and energy data, guides a student model to extract these prosodic features directly from the soft speech units. This allows the model to learn the "melody" of speech implicitly, avoiding the need for error-prone external pitch trackers during the conversion process.

Experimental Setup

The framework was trained and evaluated using two primary datasets:

  • VCTK: A multi-speaker English corpus comprising 30 hours of speech from 99 speakers, used for initial training.
  • ESD (Emotional Speaker Dataset): Used to fine-tune the prosody module, enabling the model to learn varied emotional and prosodic patterns.

The model's performance was benchmarked against several state-of-the-art baselines, including INVC, UUVC, SpeechSplit2.0, and PMVC, across both "Many-to-Many" and "Zero-Shot" (unseen speaker) scenarios.

Results and Discussion

SAVC demonstrated superior performance across both subjective and objective metrics:

  • Subjective Quality: In Many-to-Many conversion, SAVC achieved a Mean Opinion Score (MOS) of 3.89, significantly outperforming the best baseline (3.56).
  • Timbre Similarity: The model excelled in Zero-Shot scenarios, achieving a Timbre Similarity Score (TSS) of 3.87, compared to 3.55 for the nearest competitor.
  • Acoustic Fidelity: SAVC recorded the lowest Mel-Cepstral Distortion (MCD) of 5.19, indicating high fidelity to the target speaker's acoustic characteristics.
  • Intelligibility: Despite the aggressive adversarial training, the Character Error Rate (CER) remained low at 10.2%, confirming that linguistic content was preserved.

Conclusion

SAVC represents a significant step forward in expressive voice conversion. By combining soft speech units with adversarial style augmentation and implicit prosody distillation, the framework achieves a high degree of disentanglement without sacrificing speech quality.

Future work will focus on reducing the computational complexity of the multi-stage training process and decreasing reliance on pre-trained vocoders. Additionally, the researchers aim to test the framework's efficacy on tonal languages, such as Mandarin, where pitch serves a critical linguistic function.

Summary updated

Abstract

Voice conversion is the task to transform voice characteristics of source speech while preserving content information. Nowadays, self-supervised representation learning models are increasingly utilized in content extraction. However, in these representations, a lot of hidden speaker information leads to timbre leakage while the prosodic information of hidden units lacks use. To address these issues, we propose a novel framework for expressive voice conversion called “SAVC” based on soft speech units from HuBert-soft. Taking soft speech units as input, we design an attribute encoder to extract content and prosody features respectively. Specifically, we first introduce statistic perturbation imposed by adversarial style augmentation to eliminate speaker information. Then the prosody is implicitly modeled on soft speech units with knowledge distillation. Experiment results show that the intelligibility and naturalness of converted speech outperform previous work.
BibTeX
@inproceedings{deng2024learning,
  author = {Yimin Deng and Jianzong Wang and Xulong Zhang and Ning Cheng and Jing Xiao},
  title = {{Learning Expressive Disentangled Speech Representations with Soft Speech Units and Adversarial Style Augmentation}},
  booktitle = {IJCNN},
  year = {2024},
  url = {https://openreview.net/forum?id=1t9vTidqzA},
}
Figure