Abstract
While modern voice conversion (VC) frameworks like StarGANv2-VC have achieved high fidelity in mimicking target speaker identities, they frequently suffer from "emotion leakage," where the emotional state of the target reference audio overrides the affective state of the source speaker. This paper introduces StarGAN-VC++, an enhanced framework designed to disentangle speaker identity from emotional expression. By incorporating a deep emotion embedding extractor, augmented style reconstruction losses, and conversion-invariant feature preservation, StarGAN-VC++ successfully preserves the source speaker's emotion while accurately adopting the target's vocal identity.
Introduction
Voice conversion (VC) is the task of modifying the non-linguistic information of a source utterance—specifically the speaker's identity—while maintaining the linguistic content. However, speech is multi-dimensional; beyond identity and content, it carries significant affective information. Traditional many-to-many VC models often treat "style" as a monolithic variable. In practice, this leads to a phenomenon known as emotion leakage: if a user converts a happy source clip using a sad target reference, the resulting audio often sounds sad, losing the original intent of the speaker.
StarGAN-VC++ addresses this by treating emotion as a distinct component that must be preserved independently of the target speaker's style. This approach ensures that the "how" of the speech (emotion) remains constant even as the "who" (identity) changes.
Methodology
The researchers proposed three primary interventions to the StarGANv2-VC architecture to achieve emotional disentanglement.
1. Deep Emotion Embedding Extractor ($C_{emo}$)
To provide the model with a clear understanding of affect, the authors developed a "black-box" emotion extractor. This was achieved through a two-stage process: first training a model specifically for emotion conversion, then repurposing its internal layers to generate a 64-dimensional latent representation of emotion. This embedding is designed to be independent of speaker identity, providing a pure signal of the speaker's emotional state.
2. Emotion Supervision and Invariant Losses
Two new loss functions were introduced to guide the generator:
- Emotion Supervision Loss ($L_{demo}$): This ensures the deep emotion embedding of the converted speech matches that of the source speech, forcing the generator to prioritize the preservation of the source's affective state.
- Conversion Invariant Feature Preservation Loss ($L_{inv}$): Applied to the encoder, this loss ensures that latent features—both content and emotion—remain consistent between the source and the output before the decoder applies the target speaker's identity.
3. Augmented Style Reconstruction
To prevent the style encoder from inadvertently capturing transient emotional cues, the training process was modified to use two different reference utterances from the same target speaker. By requiring the model to identify commonalities between two distinct clips of the same person, the system learns to focus on permanent speaker traits (identity) rather than the specific mood or prosody of a single recording.
Experimental Setup
The model was evaluated using a diverse set of datasets to ensure robustness across genders and accents:
- ESD (Emotional Speech Database): The primary source for emotional evaluation, featuring five distinct emotions.
- VCTK (Voice Cloning Toolkit): Used to evaluate identity conversion across 109 speakers.
- RAVDESS & CREMA-D: Professional actor-based datasets used for training the underlying emotion classifiers and vocoders.
Results and Discussion
StarGAN-VC++ demonstrated significant improvements over the baseline StarGANv2-VC across both objective and subjective measures.
Objective Metrics
- Emotion Accuracy (ACCsvm): The accuracy of preserving the source emotion increased from 25.9% in the baseline to 52.7% in the proposed model.
- Pitch Correlation (PCC): The model showed a statistically significant improvement in maintaining the "melody" of the source's emotion, indicating better prosodic preservation.
- Content Integrity: Interestingly, the Character Error Rate (CER) improved slightly (from 3.5% to 3.2%), suggesting that better disentanglement of emotion actually aids the model in maintaining linguistic clarity.
Subjective Evaluation
In listening tests, users consistently identified the converted audio's emotion as matching the source rather than the target reference. This confirms that the model successfully mitigated the emotion leakage problem that plagues previous iterations of StarGAN-VC.
Key Insights
The success of StarGAN-VC++ highlights that "style" in voice conversion is a "dirty" variable—a mixture of identity, prosody, and emotion. The use of multi-reference training acts as a form of triangulation, allowing the model to isolate identity by finding the common denominator between different clips of the same speaker.
However, challenges remain. The emotion of "surprise" proved particularly difficult to preserve, often yielding 0% accuracy in specific tests. This is likely due to the fact that surprise involves rapid, extreme fluctuations in pitch that the model may still confuse with speaker-specific style or noise.
Conclusion
StarGAN-VC++ represents a significant step forward in making voice conversion more expressive and faithful to the original speaker's intent. By explicitly supervising emotion through deep embeddings and multi-reference style training, the framework achieves a superior balance between identity conversion and affective preservation. Future work will likely explore continuous Valence-Arousal models to capture more nuanced emotional states and move toward truly zero-shot emotion preservation.
Summary updated