Abstract
Emotional Voice Conversion (VC) is a technique for converting speech from one emotional state to another while preserving linguistic information and speaker identity. Traditional approaches often require parallel data and temporal alignment, which limits their practical application. This paper investigates the efficacy of using Cycle-consistent Generative Adversarial Networks (CycleGAN) for non-parallel emotional VC tasks. By incorporating two discriminators and an emotion classifier, the CycleGAN framework learns to map emotional domains without the need for frame-to-frame alignment. Objective and subjective evaluations confirm that CycleGAN models effectively alter emotional prosody while maintaining naturalness, outperforming traditional deep learning baselines in human perception tests.
Introduction
Voice Conversion (VC) typically focuses on changing speaker identity. Emotional VC is a more complex subset where the goal is to modify the emotional state of the speaker (e.g., from neutral to angry) while keeping the speaker's identity and the spoken content constant.
Historically, this task required parallel data—recordings of the same sentence spoken by the same person in different emotional states—which were then aligned using Dynamic Time Warping (DTW). This requirement is a significant bottleneck due to the scarcity of high-quality parallel emotional datasets. Furthermore, traditional models relying on L1 or L2 loss often suffer from "over-smoothing," resulting in muffled or robotic speech. This paper proposes a CycleGAN-based framework to overcome these limitations by learning emotional mappings in a non-parallel fashion.
Proposed Method
The authors employ a CycleGAN framework, originally designed for image-to-image style transfer, and adapt it for emotional speech features.
Architecture
The system integrates three primary components to ensure naturalness and emotional accuracy:
- Generators ($G_{AB}, G_{BA}$): These learn the bidirectional mapping between two emotional domains.
- Discriminators ($D_A, D_B$): These distinguish between natural and converted speech, pushing the generators to produce realistic outputs.
- Emotion Classifier ($C$): This component ensures that the converted speech effectively conveys the target emotion.
Feature Engineering and Loss Functions
To capture the nuances of emotion, the model processes both spectral and prosodic features. Mel-cepstral coefficients (MCCs) represent the timbre, while the Continuous Wavelet Transform (CWT) is used to model F0 (pitch) and energy contours across multiple time scales. This multi-scale analysis is crucial for capturing the "melody" of emotional speech.
The model is optimized using a combination of Adversarial Loss (for naturalness), Cycle-consistency Loss (to preserve linguistic content and speaker identity), and Emotion Classification Loss (to ensure the target emotion is achieved).
Experimental Evaluation
The researchers used the CASIA Chinese Emotional Corpus, focusing on Neutral, Sad, and Angry states. Although the dataset contains parallel recordings, the training process simulated a non-parallel environment by randomly pairing source and target utterances.
Key Results
While traditional DBLSTM (Deep Bidirectional LSTM) baselines often achieve lower error scores in objective metrics like Mel-cepstral Distortion (MCD)—as they are explicitly trained to minimize these distances—they often fall short in subjective quality.
In human listening tests, the CycleGAN models significantly outperformed the DBLSTM baseline. As shown in the perception results, for "Neutral to Sad" conversion, CycleGAN-2 achieved a 65.6% accuracy in emotion recognition by listeners, compared to 45.2% for the DBLSTM-1. Furthermore, listeners consistently rated the CycleGAN outputs as more natural, validating the use of adversarial training to avoid the over-smoothing issues of traditional models.
Conclusion
This study demonstrates that CycleGANs are highly effective for emotional voice conversion, particularly in scenarios where parallel data is unavailable. By combining adversarial learning with multi-scale prosody modeling via CWT, the proposed system produces speech that is both natural and emotionally convincing. Future research will explore expanding the model to a wider range of emotions and testing on fully non-parallel datasets with varying linguistic content.
Summary updated