Emotional Voice Conversion With Cycle-consistent Adversarial Network

Songxiang Liu, Yuewen Cao, Helen MengarXiv 2020 · DOI · arXiv

Abstract

Emotional Voice Conversion (VC) is a technique for converting speech from one emotional state to another while preserving linguistic information and speaker identity. Traditional approaches often require parallel data and temporal alignment, which limits their practical application. This paper investigates the efficacy of using Cycle-consistent Generative Adversarial Networks (CycleGAN) for non-parallel emotional VC tasks. By incorporating two discriminators and an emotion classifier, the CycleGAN framework learns to map emotional domains without the need for frame-to-frame alignment. Objective and subjective evaluations confirm that CycleGAN models effectively alter emotional prosody while maintaining naturalness, outperforming traditional deep learning baselines in human perception tests.

Introduction

Voice Conversion (VC) typically focuses on changing speaker identity. Emotional VC is a more complex subset where the goal is to modify the emotional state of the speaker (e.g., from neutral to angry) while keeping the speaker's identity and the spoken content constant.

Historically, this task required parallel data—recordings of the same sentence spoken by the same person in different emotional states—which were then aligned using Dynamic Time Warping (DTW). This requirement is a significant bottleneck due to the scarcity of high-quality parallel emotional datasets. Furthermore, traditional models relying on L1 or L2 loss often suffer from "over-smoothing," resulting in muffled or robotic speech. This paper proposes a CycleGAN-based framework to overcome these limitations by learning emotional mappings in a non-parallel fashion.

Proposed Method

The authors employ a CycleGAN framework, originally designed for image-to-image style transfer, and adapt it for emotional speech features.

Architecture

The system integrates three primary components to ensure naturalness and emotional accuracy:

  1. Generators ($G_{AB}, G_{BA}$): These learn the bidirectional mapping between two emotional domains.
  2. Discriminators ($D_A, D_B$): These distinguish between natural and converted speech, pushing the generators to produce realistic outputs.
  3. Emotion Classifier ($C$): This component ensures that the converted speech effectively conveys the target emotion.
The proposed CycleGAN architecture for emotional voice conversion, featuring generators, discriminators, and an emotion classifier.

Feature Engineering and Loss Functions

To capture the nuances of emotion, the model processes both spectral and prosodic features. Mel-cepstral coefficients (MCCs) represent the timbre, while the Continuous Wavelet Transform (CWT) is used to model F0 (pitch) and energy contours across multiple time scales. This multi-scale analysis is crucial for capturing the "melody" of emotional speech.

The model is optimized using a combination of Adversarial Loss (for naturalness), Cycle-consistency Loss (to preserve linguistic content and speaker identity), and Emotion Classification Loss (to ensure the target emotion is achieved).

Experimental Evaluation

The researchers used the CASIA Chinese Emotional Corpus, focusing on Neutral, Sad, and Angry states. Although the dataset contains parallel recordings, the training process simulated a non-parallel environment by randomly pairing source and target utterances.

Key Results

While traditional DBLSTM (Deep Bidirectional LSTM) baselines often achieve lower error scores in objective metrics like Mel-cepstral Distortion (MCD)—as they are explicitly trained to minimize these distances—they often fall short in subjective quality.

Subjective evaluation results showing the perception accuracy of different models across target emotions.

In human listening tests, the CycleGAN models significantly outperformed the DBLSTM baseline. As shown in the perception results, for "Neutral to Sad" conversion, CycleGAN-2 achieved a 65.6% accuracy in emotion recognition by listeners, compared to 45.2% for the DBLSTM-1. Furthermore, listeners consistently rated the CycleGAN outputs as more natural, validating the use of adversarial training to avoid the over-smoothing issues of traditional models.

Conclusion

This study demonstrates that CycleGANs are highly effective for emotional voice conversion, particularly in scenarios where parallel data is unavailable. By combining adversarial learning with multi-scale prosody modeling via CWT, the proposed system produces speech that is both natural and emotionally convincing. Future research will explore expanding the model to a wider range of emotions and testing on fully non-parallel datasets with varying linguistic content.

Summary updated

Abstract

Emotional Voice Conversion, or emotional VC, is a technique of converting speech from one emotion state into another one, keeping the basic linguistic information and speaker identity. Previous approaches for emotional VC need parallel data and use dynamic time warping (DTW) method to temporally align the source-target speech parameters. These approaches often define a minimum generation loss as the objective function, such as L1 or L2 loss, to learn model parameters. Recently, cycle-consistent generative adversarial networks (CycleGAN) have been used successfully for non-parallel VC. This paper investigates the efficacy of using CycleGAN for emotional VC tasks. Rather than attempting to learn a mapping between parallel training data using a frame-to-frame minimum generation loss, the CycleGAN uses two discriminators and one classifier to guide the learning process, where the discriminators aim to differentiate between the natural and converted speech and the classifier aims to classify the underlying emotion from the natural and converted speech. The training process of the CycleGAN models randomly pairs source-target speech parameters, without any temporal alignment operation. The objective and subjective evaluation results confirm the effectiveness of using CycleGAN models for emotional VC. The non-parallel training for a CycleGAN indicates its potential for non-parallel emotional VC.
BibTeX
@misc{liu2020emotional,
  author = {Songxiang Liu and Yuewen Cao and Helen Meng},
  title = {{Emotional Voice Conversion With Cycle-consistent Adversarial Network}},
  year = {2020},
  eprint = {2004.03781},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.03781},
}
Figure