Unsupervised Representation Disentanglement using Cross Domain Features and Adversarial Learning in Variational Autoencoder based Voice Conversion

Wen-Chin Huang, Hao Luo, Hsin-Te Hwang, Chen-Chou Lo, Yu-Huai Peng, Yu Tsao, Hsin-Min WangarXiv 2020 · DOI · arXiv

TL;DR

This paper enhances Variational Autoencoder (VAE)-based voice conversion by combining cross-domain acoustic features with adversarial training. By integrating Wasserstein GANs and an adversarial speaker classifier, the authors achieve superior disentanglement of linguistic content from speaker identity, resulting in converted speech with significantly higher naturalness and speaker similarity.

What Problem Does This Paper Address?

Voice Conversion (VC) aims to modify a source speaker's voice to sound like a target speaker while preserving the original linguistic content. A common approach uses Variational Autoencoders (VAEs) to decompose speech into a latent code (representing content) and a speaker code (representing identity).

However, these models often suffer from representation entanglement: the latent code frequently retains residual information about the source speaker. As illustrated in Figure 1, when this "polluted" latent code is combined with a target speaker code, the decoder produces a voice that sounds like a mix of both speakers, degrading conversion performance. Furthermore, VAEs often produce "blurry" or over-smoothed spectral features, leading to muffled-sounding speech.

Fig. 1: Illustration of how entangled latent representation affects the conversion performance in a general VAE-VC framework. The residual source speaker information in the latent code will be mixed with the given target speaker code, resulting in a mixed speaker identity in the converted feature. Thus, the performance might be harmed.

Methods

The authors build upon their previous Cross-Domain VAE (CDVAE-VC) framework, which simultaneously models two types of spectral features: high-dimensional STRAIGHT spectra (SPs) and human-perception-related mel-cepstral coefficients (MCCs).

To address the remaining challenges, they introduce two adversarial components:

  1. Generative Adversarial Networks (GANs): They incorporate a Wasserstein GAN (WGAN) objective. The VAE's decoder acts as a generator, while a discriminator learns to distinguish between real spectral features and those reconstructed by the model. This forces the decoder to produce sharper, more realistic spectra, overcoming the over-smoothing issue.
  2. Adversarial Speaker Classifier (CLS): To explicitly strip speaker information from the latent code, they add a classifier that tries to predict the speaker's identity from the latent representation. The encoder is then trained to "fool" this classifier, effectively projecting away speaker-dependent traits and leaving only speaker-independent phonetic information.

The training process is a sophisticated three-phase schedule, as shown in Figure 4, involving pre-training the VAE and classifier before moving into a joint adversarial min-max game.

Fig. 4: Illustration of the training procedure of our proposed CDVAE-CLS-GAN model. Phase 1: A CDVAE is trained. Phase 2: The latent codes are used to train the CLS. Phase 3-A and 3-B: The encoders, decoders and the CLS, discriminators are trained in an alternating order.

Datasets

The researchers evaluated their models using the Voice Conversion Challenge (VCC) 2018 dataset. This benchmark contains recordings from 12 professional US English speakers. The data was split into 70 utterances for training, 11 for validation, and 35 for testing per speaker.

Key Insights and Ideas

  • The Power of Cross-Domain Features: Modeling SPs and MCCs together (Figure 3) provides a stronger constraint on the latent space than using either alone, facilitating better disentanglement from the start.
  • GANs as Disentanglers: Interestingly, the authors found that GANs don't just improve output quality; they also indirectly help the encoder learn better latent representations.
  • Quantifying Disentanglement (DEM): The paper utilizes a novel metric called the Disentanglement Measure (DEM). It calculates the cosine similarity between latent codes of parallel sentences (the same sentence spoken by different people). An ideal encoder should produce nearly identical latent codes for the same content, regardless of the speaker.
  • Statistics Matching: The adversarial training allows the model to match the Global Variance (GV) and Modulation Spectrum (MS) of real speech much more closely than standard VAEs (see Figures 5 and 6), potentially eliminating the need for traditional post-filtering techniques.
Fig. 5: Global variance curves showing how GAN-based models (red/brown) better match the target statistics.

Key Results

  • Subjective Naturalness: In Mean Opinion Score (MOS) tests, the proposed CDVAE-CLS-GAN models significantly outperformed the baseline CDVAE. For example, adding CLS and GAN to the MCC path raised the average MOS from 2.40 to 3.15.
  • Speaker Similarity: The adversarial speaker classifier (CLS) was highly effective. The CDVAE-CLS-GAN model achieved the highest similarity scores in listening tests, confirming that eliminating source speaker identity from the latent code is key to successful conversion.
  • Objective Metrics: While Mel-Cepstrum Distortion (MCD) sometimes increased with adversarial training (a known phenomenon where lower distortion doesn't always mean better sound), the improvements in GV and MS statistics correlated strongly with better perceptual quality.

Limitations and Future Work

  • Feature Engineering: The current system still relies on vocoder-extracted features (SPs, MCCs, $F_0$). The authors suggest exploring "rawer" features like the magnitude spectrum or different hand-crafted features like line-spectral pairs.
  • Latent Dimensioning: The choice of a 16-dimensional latent space was empirical; finding an algorithmic way to determine the optimal dimension for disentanglement remains an open question.
  • Scaling Up: The current work focuses on a closed set of speakers. A natural next step is generalizing this framework to many-to-one or zero-shot voice conversion, where the model can extract speaker-invariant representations from speakers it has never seen during training.

Summary updated

Abstract

An effective approach for voice conversion (VC) is to disentangle linguistic content from other components in the speech signal. The effectiveness of variational autoencoder (VAE) based VC (VAE-VC), for instance, strongly relies on this principle. In our prior work, we proposed a cross-domain VAE-VC (CDVAE-VC) framework, which utilized acoustic features of different properties, to improve the performance of VAE-VC. We believed that the success came from more disentangled latent representations. In this paper, we extend the CDVAE-VC framework by incorporating the concept of adversarial learning, in order to further increase the degree of disentanglement, thereby improving the quality and similarity of converted speech. More specifically, we first investigate the effectiveness of incorporating the generative adversarial networks (GANs) with CDVAE-VC. Then, we consider the concept of domain adversarial training and add an explicit constraint to the latent representation, realized by a speaker classifier, to explicitly eliminate the speaker information that resides in the latent code. Experimental results confirm that the degree of disentanglement of the learned latent representation can be enhanced by both GANs and the speaker classifier. Meanwhile, subjective evaluation results in terms of quality and similarity scores demonstrate the effectiveness of our proposed methods.
BibTeX
@misc{huang2020unsupervised,
  author = {Wen-Chin Huang and Hao Luo and Hsin-Te Hwang and Chen-Chou Lo and Yu-Huai Peng and Yu Tsao and Hsin-Min Wang},
  title = {{Unsupervised Representation Disentanglement using Cross Domain Features and Adversarial Learning in Variational Autoencoder based Voice Conversion}},
  year = {2020},
  eprint = {2001.07849},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.07849},
}

Loading related content...

Figure