PMVC: Data Augmentation-Based Prosody Modeling for Expressive Voice Conversion

Yimin Deng, Huaizhen Tang, Xulong Zhang, Jianzong Wang, Ning Cheng, Jing Xiao

Abstract

Voice conversion (VC) serves as a style transfer task for speech, aiming to transform a source speaker's voice into a target identity while maintaining linguistic integrity. While modern VC models effectively replicate timbre, they often struggle with prosody—the rhythm, pauses, and pitch contours that define expressive speech. This paper introduces PMVC, a novel framework designed for expressive voice conversion without the need for text transcriptions. By employing a unique "Random Prosody" data augmentation algorithm and an adversarial "mask and predict" mechanism, PMVC successfully disentangles content, timbre, and prosodic information. Experimental results on the AIShell-3 corpus demonstrate that PMVC significantly improves the naturalness and similarity of converted speech compared to existing baselines.

Introduction

In the field of speech synthesis, voice conversion is frequently described as "style transfer for audio." While the technology has matured to the point of achieving high timbre similarity, the challenge of capturing prosody remains. Most existing systems simply map a target speaker's timbre onto the source speaker's original rhythm, often resulting in speech that sounds unnatural or lacks the target's characteristic expressive nuances.

Furthermore, many prosody-aware models rely on text transcriptions or phoneme alignments, creating a bottleneck for training on the vast quantities of unlabelled audio data available. PMVC addresses these challenges by providing a completely "text-free" framework for modeling prosody, ensuring that the converted output captures not just who is speaking, but how they speak.

Methodology

Disentangled Representations

The PMVC framework decomposes speech into three independent components:

  1. Content ($C_x$): The linguistic information.
  2. Timbre ($S_x$): The global speaker identity.
  3. Prosody ($P_x$): The local rhythmic and melodic features.

To achieve this separation, the model utilizes Instance Normalization (IN) within the encoder to filter out global speaker characteristics, effectively isolating time-varying features like content and prosody.

Random Prosody Data Augmentation

The core innovation of PMVC is the Random Prosody Algorithm. To teach the model to isolate prosody, the authors developed a strategy that modifies the rhythm of a speech segment without altering its content or total duration. The algorithm splits a segment into uniform pieces and randomly stretches some while shortening others. Because the total length remains constant, the model can perform frame-level contrastive learning, forcing the content embeddings of the original and augmented audio to remain identical while their prosody embeddings diverge.

Adversarial Information Bottleneck

To prevent the prosody vector from inadvertently capturing linguistic data (content leakage), PMVC employs a Gradient Reversal Layer (GRL). A "Content Predictor" attempts to decode words from the prosody vector; simultaneously, the encoder is trained to make this prediction impossible. This adversarial setup acts as an information bottleneck, ensuring the prosody representation is "pure" and contains only rhythmic information.

Experimental Results

Dataset and Setup

The model was evaluated using the AIShell-3 corpus, a large-scale Mandarin Chinese dataset containing 85 hours of speech from 218 speakers. The evaluation covered both many-to-many conversion (seen speakers) and zero-shot conversion (unseen speakers).

Performance Metrics

PMVC outperformed several competitive baselines, including AutoVC, INVC, and SpeechFlow, across both objective and subjective measures:

  • Objective Quality: PMVC achieved a Mel-Cepstral Distortion (MCD) of 6.06, a significant improvement over SpeechFlow (6.67) and INVC (9.18).
  • Subjective Naturalness: In Mean Opinion Score (MOS) evaluations, PMVC received a score of 3.64, the highest among all tested models.
  • Similarity: Voice Similarity Scores (VSS) indicated that PMVC more accurately captured the target's prosodic style and timbre.
  • Efficiency: The model is relatively lightweight, containing 20.8M parameters—approximately 35% smaller than architectural variants that use separate encoders for content and prosody.

Discussion and Future Work

While PMVC represents a significant step forward in text-free expressive voice conversion, several areas for improvement remain. The current Random Prosody Algorithm uses uniform-length segments; the authors suggest that variable-length segments might better capture complex human rhythmic patterns. Additionally, while the model excels in Mandarin, further research is required to determine how these prosodic features transfer across different language families, such as from tonal to non-tonal languages.

Conclusion

PMVC demonstrates that robust prosody modeling is achievable without text transcriptions through clever data augmentation and adversarial training. By successfully disentangling the "how" from the "what" in speech, PMVC moves voice conversion closer to truly human-like expressivity, offering a scalable solution for high-quality speech synthesis using unlabelled data.

Summary updated

Abstract

Voice conversion as the style transfer task applied to speech, refers to converting one person's speech into a new speech that sounds like another person's. Up to now, there has been a lot of research devoted to better implementation of VC tasks. However, a good voice conversion model should not only match the timbre information of the target speaker, but also expressive information such as prosody, pace, pause, etc. In this context, prosody modeling is crucial for achieving expressive voice conversion that sounds natural and convincing. Unfortunately, prosody modeling is important but challenging, especially without text transcriptions. In this paper, we firstly propose a novel voice conversion framework named 'PMVC', which effectively separates and models the content, timbre, and prosodic information from the speech without text transcriptions. Specially, we introduce a new speech augmentation algorithm for robust prosody extraction. And building upon this, mask and predict mechanism is applied in the disentanglement of prosody and content information. The experimental results on the AIShell-3 corpus supports our improvement of naturalness and similarity of converted speech.
BibTeX
@inproceedings{deng2023pmvc,
  author = {Yimin Deng and Huaizhen Tang and Xulong Zhang and Jianzong Wang and Ning Cheng and Jing Xiao},
  title = {{PMVC: Data Augmentation-Based Prosody Modeling for Expressive Voice Conversion}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3613800},
}
Figure