PMVC: Data Augmentation-Based Prosody Modeling for Expressive Voice Conversion

Yimin Deng, Huaizhen Tang, Xulong Zhang, Jianzong Wang, Ning Cheng, Jing Xiao

TL;DR

PMVC is a novel voice conversion framework that captures not just who is speaking (timbre) but how they speak (prosody) without requiring any text transcriptions. By using a clever "Random Prosody" data augmentation algorithm and adversarial training, the model successfully disentangles speech into independent components of content, speaker identity, and rhythm, leading to more natural and expressive voice swaps.

What Problem Does This Paper Address?

Voice Conversion (VC) is often described as "style transfer for speech." While modern models are excellent at mimicking a target speaker's timbre (the unique "texture" of their voice), they often fail to capture prosody—the rhythm, pauses, and pitch contours that make speech sound human and expressive.

Most existing VC systems simply "paste" the target speaker's voice onto the source speaker's original rhythm. Furthermore, many models that do attempt to model prosody rely on text transcriptions or phoneme alignments. This creates a bottleneck: you can't train or use these models on the vast amounts of unlabelled audio data available. PMVC aims to achieve expressive, prosody-aware conversion in a completely "text-free" manner.

Methods

The authors introduce a framework that decomposes speech into three disentangled representations: Content ($C_x$), Timbre ($S_x$), and Prosody ($P_x$). The core innovations include:

  • Random Prosody Algorithm (Data Augmentation): To teach the model what "prosody" is, the authors created a new augmentation strategy. They take a speech segment, split it into uniform pieces, and then randomly stretch some pieces while shortening others so the total length remains the same. This creates a version of the audio with the same words (content) and voice (timbre) but a different rhythm (prosody).
  • Contrastive Learning: The model is trained to ensure that the content embeddings of the original and augmented audio are nearly identical, while their prosody embeddings are forced to be different.
  • Mask and Predict (Adversarial Training): To prevent the model from "cheating" by hiding word information inside the prosody vector, they use a Gradient Reversal Layer (GRL). A "Content Predictor" tries to guess the words from the prosody vector; the encoder is simultaneously trained to make this task impossible, effectively scrubbing content info from the prosody channel.
  • Instance Normalization (IN): The encoder uses IN to remove global speaker characteristics, helping to isolate the time-varying features (content and prosody) from the static ones (timbre).

Datasets

The model was evaluated using the AIShell-3 corpus, a large-scale Mandarin Chinese dataset containing roughly 85 hours of speech from 218 native speakers. The authors tested the model on both "many-to-many" tasks (speakers seen during training) and "zero-shot" tasks (converting voices of speakers the model has never encountered).

Key Insights and Ideas

  • Prosody is Local, Timbre is Global: The paper builds on the intuition that speaker identity is a global, static trait of an utterance, while content and prosody change frame-by-frame. Using Instance Normalization allows the model to "filter" out the global identity.
  • The "Constant Length" Trick: Unlike previous random resampling methods that change the total duration of audio, PMVC’s augmentation keeps the duration fixed. This allows for a direct, frame-level comparison between original and augmented features, making the contrastive loss much more stable and effective.
  • Information Theory Constraints: The use of the GRL acts as an information bottleneck. It forces the prosody embedding to be "pure"—containing only the "how" and none of the "what."

Key Results

PMVC outperformed several strong baselines, including AutoVC, INVC, and SpeechFlow, in both objective and subjective metrics:

  • Objective Quality: PMVC achieved a Mel-Cepstral Distortion (MCD) of 6.06, significantly lower (better) than SpeechFlow (6.67) and INVC (9.18).
  • Naturalness: In Mean Opinion Score (MOS) tests, humans rated PMVC at 3.64, the highest among the tested models.
  • Similarity: The Voice Similarity Score (VSS) showed that PMVC was better at capturing both the target's timbre and their prosodic style compared to baselines.
  • Efficiency: Despite its performance, the PMVC model is relatively compact (20.8M parameters), making it nearly 35% smaller than its variant PMVC_t, which used separate encoders for content and prosody.

Limitations and Future Work

  • Segment Granularity: The Random Prosody Algorithm currently uses uniform-length segments. The authors suggest that using variable-length segments might capture even more complex rhythmic patterns.
  • Language Specificity: While the results on Mandarin are impressive, the paper does not explore how well these prosodic features transfer across different languages (e.g., from a tonal language like Mandarin to a non-tonal one like English).
  • Pretrained Dependency: The system still relies on a pretrained speaker encoder (using GE2E loss) to provide the timbre embedding, meaning it isn't entirely trained from scratch in one go.

Summary updated

Abstract

Voice conversion as the style transfer task applied to speech, refers to converting one person's speech into a new speech that sounds like another person's. Up to now, there has been a lot of research devoted to better implementation of VC tasks. However, a good voice conversion model should not only match the timbre information of the target speaker, but also expressive information such as prosody, pace, pause, etc. In this context, prosody modeling is crucial for achieving expressive voice conversion that sounds natural and convincing. Unfortunately, prosody modeling is important but challenging, especially without text transcriptions. In this paper, we firstly propose a novel voice conversion framework named 'PMVC', which effectively separates and models the content, timbre, and prosodic information from the speech without text transcriptions. Specially, we introduce a new speech augmentation algorithm for robust prosody extraction. And building upon this, mask and predict mechanism is applied in the disentanglement of prosody and content information. The experimental results on the AIShell-3 corpus supports our improvement of naturalness and similarity of converted speech.
BibTeX
@inproceedings{deng2023pmvc,
  author = {Yimin Deng and Huaizhen Tang and Xulong Zhang and Jianzong Wang and Ning Cheng and Jing Xiao},
  title = {{PMVC: Data Augmentation-Based Prosody Modeling for Expressive Voice Conversion}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3613800},
}

Loading related content...

Figure