One-Shot Voice Conversion with Disentangled Representations by Leveraging Phonetic Posteriorgrams

Seyed Hamidreza Mohammadi, Taehwan KimInterspeech 2019

TL;DR

This paper introduces a novel one-shot voice conversion (VC) system that disentangles speaker identity from linguistic content. By leveraging pre-trained phonetic posteriorgrams (PPGs) from an ASR model, their approach significantly simplifies training and achieves superior speaker similarity compared to existing methods, even with minimal target speaker data.

What Problem Does This Paper Address?

Voice conversion (VC) aims to transform a source speaker's voice into a target speaker's voice while preserving the linguistic content. Traditional VC methods often require large amounts of parallel speech data (the same sentences spoken by both source and target speakers), which is expensive and difficult to obtain. While non-parallel methods have emerged, they still typically need extensive training data for each speaker pair or knowledge of the speaker's identity during training. The challenge this paper tackles is enabling one-shot VC – converting speech to a new speaker's voice using only one or a few examples of that speaker – without requiring parallel data or extensive speaker-specific training.

Methods

The core innovation lies in their proposed model architecture and the use of phonetic posteriorgrams (PPGs). The model consists of two main components:

  • RNN Encoder: This network takes acoustic features of the source speech as input and learns a "speaker embedding" vector, representing the unique characteristics of the speaker.
  • RNN Decoder: This network takes the learned speaker embedding and, crucially, phonetic posteriorgrams (PPGs) as input to reconstruct the acoustic features.

The key insight is that PPGs, which are the output probabilities of phonetic states from a well-trained Automatic Speech Recognition (ASR) model, already encode the linguistic content in a speaker-independent manner. By using these pre-computed PPGs, the model can focus solely on learning the speaker embedding, significantly simplifying the disentanglement process. This avoids the need for complex adversarial training or hierarchical structures often employed in other disentanglement methods. For voice conversion, they found that adding a "difference vector" between the average source and target speaker embeddings to the source embedding before decoding yielded better results than simply replacing the source embedding with the target's.

Datasets

  • Training: TIMIT corpus (multi-speaker speech corpus) was used for training both their model and the baseline.
  • Objective Testing: CMU-arctic voices (BDL, SLT, RMS, CLB) were used for objective evaluation, requiring parallel data.
  • Phonetic Posteriorgrams: Kaldi ASR toolkit was used to compute PPGs from the Librispeech corpus.

Key Insights and Ideas

  • PPGs as Linguistic Anchors: Leveraging pre-trained PPGs from ASR models is a highly effective way to provide a robust, speaker-independent linguistic context, simplifying the disentanglement task.
  • Focus on Speaker Identity: By offloading the linguistic encoding to PPGs, the model can dedicate its learning capacity to disentangling and learning speaker-specific representations more efficiently.
  • Simplicity is Key: Their model is simpler than adversarial or hierarchical approaches, leading to faster training times (hours vs. days for FHVAE).
  • Difference Vector for Conversion: Using a difference vector between source and target speaker embeddings for conversion, rather than direct replacement, proved more effective, suggesting that speaker embeddings might not be perfectly disentangled.

Key Results

  • Subjective Evaluation: The proposed model achieved significantly better speaker similarity compared to the baseline (FHVAE), with positive scores favoring their approach.
  • Speech Quality: While not as pronounced as similarity, their model showed a statistically significant preference in the F2M (female-to-male) conversion condition.
  • Training Efficiency: Their model trains in approximately 5 hours, a substantial improvement over the 2 days required for the FHVAE baseline.
  • Visualization: Speaker embeddings visualized via PCA showed better separation and distribution for the proposed model compared to FHVAE, indicating more robust speaker disentanglement.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their model shows improved similarity, the speech quality improvements are less dramatic and statistically significant only in specific conditions. The use of a difference vector suggests that perfect disentanglement might not be achieved.
  • Future Work: Not explicitly detailed, but the success of this approach opens avenues for exploring more complex conversion tasks (e.g., style transfer) by incorporating additional disentangled representations, or further optimizing the speaker embedding extraction for even higher quality and similarity.

Summary updated

BibTeX
@inproceedings{mohammadi2019one,
  author = {Seyed Hamidreza Mohammadi and Taehwan Kim},
  title = {{One-Shot Voice Conversion with Disentangled Representations by Leveraging Phonetic Posteriorgrams}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure