Introduction
Voice Conversion (VC) is the process of modifying the speaker identity (timbre) of a source utterance while preserving its linguistic content and prosodic nuances. Despite recent advancements in self-supervised learning (SSL), achieving high-fidelity conversion remains a challenge due to the difficulty of perfectly disentangling speech components. This paper introduces SAVC (Soft speech units and Adversarial style augmentation for Voice Conversion), a framework designed to produce expressive, high-quality speech by effectively separating content, timbre, and prosody.
The Problem: Timbre Leakage and Prosody Neglect
Modern VC systems often rely on SSL models like HuBERT to extract content representations. However, these systems frequently encounter two primary obstacles:
- Timbre Leakage: SSL representations are not purely linguistic; they often retain residual information about the source speaker's identity. This leads to "leakage," where the converted audio sounds like a hybrid of the source and target speakers, undermining the conversion's authenticity.
- Prosody Neglect: Many models prioritize content and timbre at the expense of prosody—the rhythm, pitch, and energy that convey emotion and naturalness. This often results in speech that sounds robotic or fails to capture the expressive intent of the original speaker.
While some researchers use manual signal processing or information bottlenecks to address these issues, these methods can inadvertently degrade overall speech quality.
The SAVC Framework
SAVC addresses these challenges through three core innovations: Soft Speech Units, Adversarial Style Augmentation (ASA), and Prosody Distillation.
Soft Speech Units
Rather than using discrete, quantized speech tokens which can discard subtle linguistic details, SAVC utilizes "soft" units derived from HuBERT-Soft. These units provide a continuous representation that retains more nuance than discrete tokens while remaining significantly more speaker-independent than raw mel-spectrograms.
Adversarial Style Augmentation (ASA)
The central innovation of SAVC is the Adversarial Style Augmentation (ASA) technique. During training, the model applies dynamic statistical perturbations to the mean and standard deviation of speech features. A Gradient Reversal Layer (GRL) is then employed to force the attribute encoder to become invariant to these speaker-specific "style" statistics. This process effectively "scrubs" the source speaker's identity from the content and prosody embeddings, ensuring a cleaner separation of attributes.
Prosody Distillation
To capture expressiveness without requiring explicit pitch or energy labels during inference, SAVC employs a teacher-student distillation setup. A teacher model, trained with explicit access to pitch and energy data, guides a student model to extract these prosodic features directly from the soft speech units. This allows the model to learn the "melody" of speech implicitly, avoiding the need for error-prone external pitch trackers during the conversion process.
Experimental Setup
The framework was trained and evaluated using two primary datasets:
- VCTK: A multi-speaker English corpus comprising 30 hours of speech from 99 speakers, used for initial training.
- ESD (Emotional Speaker Dataset): Used to fine-tune the prosody module, enabling the model to learn varied emotional and prosodic patterns.
The model's performance was benchmarked against several state-of-the-art baselines, including INVC, UUVC, SpeechSplit2.0, and PMVC, across both "Many-to-Many" and "Zero-Shot" (unseen speaker) scenarios.
Results and Discussion
SAVC demonstrated superior performance across both subjective and objective metrics:
- Subjective Quality: In Many-to-Many conversion, SAVC achieved a Mean Opinion Score (MOS) of 3.89, significantly outperforming the best baseline (3.56).
- Timbre Similarity: The model excelled in Zero-Shot scenarios, achieving a Timbre Similarity Score (TSS) of 3.87, compared to 3.55 for the nearest competitor.
- Acoustic Fidelity: SAVC recorded the lowest Mel-Cepstral Distortion (MCD) of 5.19, indicating high fidelity to the target speaker's acoustic characteristics.
- Intelligibility: Despite the aggressive adversarial training, the Character Error Rate (CER) remained low at 10.2%, confirming that linguistic content was preserved.
Conclusion
SAVC represents a significant step forward in expressive voice conversion. By combining soft speech units with adversarial style augmentation and implicit prosody distillation, the framework achieves a high degree of disentanglement without sacrificing speech quality.
Future work will focus on reducing the computational complexity of the multi-stage training process and decreasing reliance on pre-trained vocoders. Additionally, the researchers aim to test the framework's efficacy on tonal languages, such as Mandarin, where pitch serves a critical linguistic function.
Summary updated