TL;DR
This paper enhances Variational Autoencoder (VAE)-based voice conversion by combining cross-domain acoustic features with adversarial training. By integrating Wasserstein GANs and an adversarial speaker classifier, the authors achieve superior disentanglement of linguistic content from speaker identity, resulting in converted speech with significantly higher naturalness and speaker similarity.
What Problem Does This Paper Address?
Voice Conversion (VC) aims to modify a source speaker's voice to sound like a target speaker while preserving the original linguistic content. A common approach uses Variational Autoencoders (VAEs) to decompose speech into a latent code (representing content) and a speaker code (representing identity).
However, these models often suffer from representation entanglement: the latent code frequently retains residual information about the source speaker. As illustrated in Figure 1, when this "polluted" latent code is combined with a target speaker code, the decoder produces a voice that sounds like a mix of both speakers, degrading conversion performance. Furthermore, VAEs often produce "blurry" or over-smoothed spectral features, leading to muffled-sounding speech.
Methods
The authors build upon their previous Cross-Domain VAE (CDVAE-VC) framework, which simultaneously models two types of spectral features: high-dimensional STRAIGHT spectra (SPs) and human-perception-related mel-cepstral coefficients (MCCs).
To address the remaining challenges, they introduce two adversarial components:
- Generative Adversarial Networks (GANs): They incorporate a Wasserstein GAN (WGAN) objective. The VAE's decoder acts as a generator, while a discriminator learns to distinguish between real spectral features and those reconstructed by the model. This forces the decoder to produce sharper, more realistic spectra, overcoming the over-smoothing issue.
- Adversarial Speaker Classifier (CLS): To explicitly strip speaker information from the latent code, they add a classifier that tries to predict the speaker's identity from the latent representation. The encoder is then trained to "fool" this classifier, effectively projecting away speaker-dependent traits and leaving only speaker-independent phonetic information.
The training process is a sophisticated three-phase schedule, as shown in Figure 4, involving pre-training the VAE and classifier before moving into a joint adversarial min-max game.
Datasets
The researchers evaluated their models using the Voice Conversion Challenge (VCC) 2018 dataset. This benchmark contains recordings from 12 professional US English speakers. The data was split into 70 utterances for training, 11 for validation, and 35 for testing per speaker.
Key Insights and Ideas
- The Power of Cross-Domain Features: Modeling SPs and MCCs together (Figure 3) provides a stronger constraint on the latent space than using either alone, facilitating better disentanglement from the start.
- GANs as Disentanglers: Interestingly, the authors found that GANs don't just improve output quality; they also indirectly help the encoder learn better latent representations.
- Quantifying Disentanglement (DEM): The paper utilizes a novel metric called the Disentanglement Measure (DEM). It calculates the cosine similarity between latent codes of parallel sentences (the same sentence spoken by different people). An ideal encoder should produce nearly identical latent codes for the same content, regardless of the speaker.
- Statistics Matching: The adversarial training allows the model to match the Global Variance (GV) and Modulation Spectrum (MS) of real speech much more closely than standard VAEs (see Figures 5 and 6), potentially eliminating the need for traditional post-filtering techniques.
Key Results
- Subjective Naturalness: In Mean Opinion Score (MOS) tests, the proposed CDVAE-CLS-GAN models significantly outperformed the baseline CDVAE. For example, adding CLS and GAN to the MCC path raised the average MOS from 2.40 to 3.15.
- Speaker Similarity: The adversarial speaker classifier (CLS) was highly effective. The CDVAE-CLS-GAN model achieved the highest similarity scores in listening tests, confirming that eliminating source speaker identity from the latent code is key to successful conversion.
- Objective Metrics: While Mel-Cepstrum Distortion (MCD) sometimes increased with adversarial training (a known phenomenon where lower distortion doesn't always mean better sound), the improvements in GV and MS statistics correlated strongly with better perceptual quality.
Limitations and Future Work
- Feature Engineering: The current system still relies on vocoder-extracted features (SPs, MCCs, $F_0$). The authors suggest exploring "rawer" features like the magnitude spectrum or different hand-crafted features like line-spectral pairs.
- Latent Dimensioning: The choice of a 16-dimensional latent space was empirical; finding an algorithmic way to determine the optimal dimension for disentanglement remains an open question.
- Scaling Up: The current work focuses on a closed set of speakers. A natural next step is generalizing this framework to many-to-one or zero-shot voice conversion, where the model can extract speaker-invariant representations from speakers it has never seen during training.
Summary updated