TL;DR
This paper introduces DiffEmotionVC, a novel diffusion-based framework for "any-to-any" emotional voice conversion (EVC). It tackles the challenge of disentangling complex emotional features from speaker and content information by employing a dual-granularity emotion encoder and an orthogonality-constrained condition encoder, achieving state-of-the-art results in speech quality and emotion recognition.
What Problem Does This Paper Address?
Emotional Voice Conversion (EVC) aims to modify the emotional tone of speech while preserving the original content and speaker's identity. While significant progress has been made, current EVC systems still struggle with several challenges. A primary issue is the inherent entanglement of emotional features with speaker characteristics and linguistic content, making it difficult to isolate and manipulate emotions effectively. Furthermore, existing methods often lack the ability to perform "any-to-any" conversion, meaning they might struggle with unseen emotions or speakers. Existing approaches, like those using mutual information disentanglement, can also suffer from training instability. This paper addresses these limitations by proposing a framework that can disentangle emotional features more robustly and achieve flexible, any-to-any EVC.
Methods
DiffEmotionVC is built upon a diffusion model, a generative approach known for producing high-quality audio. The core innovations lie in its encoder architecture and training objectives:
- Dual-Granularity Emotion Encoder: This encoder captures emotional context at two levels: utterance-level for global emotional trends and frame-level for fine-grained acoustic details. This multi-scale approach aims to provide a more comprehensive representation of emotion.
- Orthogonality-Constrained Condition Encoder: This component is crucial for disentangling features. It uses gated cross-attention to fuse emotion, speaker, and content information dynamically. Crucially, an "orthogonal loss" is applied to enforce independence between these feature representations, preventing them from interfering with each other.
- Multi-Objective Diffusion Training: The framework employs a three-phase training strategy that includes:
- Acoustic Reconstruction Fidelity: Standard diffusion model objective to ensure the generated speech sounds natural.
- Contrastive Metric Learning: This helps the model learn to distinguish between subtle differences in emotional features, improving emotion recognition accuracy.
- Perceptual Loss Optimization: This likely refers to objectives that align with human perception of speech quality and emotion.
The framework also leverages pre-trained models for speaker (Resemblyzer) and content (ContentVec) feature extraction, allowing it to focus on the challenging task of emotion disentanglement.
Datasets
- Emilia Dataset: Used for pre-training the DiffEmotionVC model. This dataset is noted for its multi-scenario data and efforts to preserve emotional authenticity.
- ESD Dataset: Used for training and testing the model. This dataset contains five emotions (neutral, sad, angry, surprise, happy) and is characterized by different arousal levels across emotions.
Key Insights and Ideas
- Dual-granularity is better than single-granularity: Capturing emotion at both broad (utterance) and fine (frame) levels leads to richer representations and better EVC performance.
- Orthogonality is key for disentanglement: Explicitly enforcing orthogonality between feature representations (emotion, speaker, content) is a powerful way to prevent feature entanglement and improve the model's ability to isolate emotional characteristics.
- Contrastive learning boosts emotion discrimination: Beyond standard classification, contrastive learning helps the model learn finer distinctions between emotions, which is crucial for accurate emotional expression.
- Diffusion models are well-suited for EVC: The paper demonstrates that diffusion models can effectively generate high-quality emotional speech and handle the complex disentanglement task.
Key Results
The proposed DiffEmotionVC framework achieved impressive results:
- Subjective Evaluation: A UTMOS score of 4.04, indicating high speech quality and naturalness.
- Objective Evaluation: 80% emotion recognition accuracy, demonstrating the effectiveness of the disentangled emotion features.
- Comparison: The paper shows competitive or superior performance compared to several state-of-the-art EVC models like CycleGan-EVC, StarGAN-EVC, Any-to-Any-EVC, and StableVC across various metrics.
Limitations and Future Work
- Limitations: The authors note that while their model can distinguish between high-arousal emotions (happy, surprise, angry), it faces some challenges. This is attributed to the potential insufficiency of natural emotional expressions in existing datasets.
- Future Work: The authors plan to focus on deeper understanding of complex emotional nuances in real-world expressions to further improve EVC. They also aim to explore multi-emotion control capabilities within their framework.