Speaker Adaption with Intuitive Prosodic Features for Statistical Parametric Speech Synthesis

Pengyu Cheng, Zhen-Hua Ling

Introduction

Modern neural text-to-speech (TTS) systems have achieved remarkable levels of naturalness, yet they remain notoriously "data-hungry." Typically, generating a high-quality digital twin of a specific voice requires 10 to 20 hours of studio-grade recordings. This requirement is a significant barrier for applications such as personalized virtual assistants or voice restoration for patients who have lost their ability to speak.

To bridge this gap, researchers use speaker adaptation (or voice cloning), where a model pre-trained on a diverse multi-speaker dataset is fine-tuned using a very small amount of data from a target individual. However, existing adaptation methods often fail to capture the "soul" of a voice—its prosody, which includes rhythm, stress, and intonation. While some systems attempt to capture these nuances using complex "latent" variables, these black-box vectors often inadvertently capture background noise or recording conditions rather than the speaker's actual vocal characteristics. This research explores whether providing AI models with explicit, human-understandable speech traits can improve the accuracy and efficiency of voice cloning.

Methodology

The researchers built their system upon Tacotron2, a standard sequence-to-sequence acoustic model. The core innovation lies in the integration of "intuitive prosodic features" directly into the synthesis pipeline. Instead of forcing the model to infer prosody from raw audio, the authors explicitly extract and provide four key metrics:

  • Pitch ($F_0$): The average fundamental frequency of the speaker.
  • Pitch Range: The variation between the speaker's highest and lowest notes.
  • Speech Rate: The average duration of phonemes, representing how fast the person speaks.
  • Energy: The average volume or intensity of the speech.

Proposed Architectures

The study proposed two distinct methods for integrating these features:

  1. UPF-Enc (Utterance-level Prosodic Features): This model utilizes a speaker encoder to generate a unique vector for each specific sentence. This approach is particularly effective for "zero-shot" scenarios where the system must adapt to a new voice based on a single reference recording.
  2. SPF-Emb (Speaker-level Prosodic Features): This model uses a speaker embedding table to represent the aggregate characteristics of a speaker across their entire dataset.

In both architectures, the four prosodic features are concatenated with the text encoder's output and the speaker vector. This combined "super-vector" is then processed by the decoder to generate Mel-spectrograms.

Experimental Setup

The models were trained and evaluated using the AISHELL-3 dataset, a large-scale Mandarin Chinese corpus.

  • Pre-training: 166 speakers were used to establish a baseline for speech synthesis.
  • Adaptation: 8 unseen speakers (4 male, 4 female) were used to test the cloning capabilities.
  • Robustness: To ensure the speaker encoder could handle diverse voices, the researchers utilized a massive auxiliary dataset of over 775,000 utterances from the aidataTang and MAGICDATA corpora.

Key Insights and Results

The research yielded several critical findings regarding the role of explicit prosody in speech synthesis:

Explicit vs. Latent Modeling

The study demonstrated that "explicit is better than latent." By providing the model with actual speech rates and pitch values, the researchers removed the guesswork inherent in Variational Autoencoders (VAEs). This led to more stable and accurate voice clones that were less susceptible to environmental noise.

The Hierarchy of Features

Through ablation studies—where features were removed one by one to test their impact—the authors identified a clear hierarchy of importance. Pitch and speech rate were found to be the most critical features for speaker identity. When these were removed, the model's performance dropped significantly more than when energy or pitch range were omitted.

Performance Metrics

The UPF-Enc model emerged as the top performer in both objective and subjective evaluations:

  • Objective: It achieved the lowest Mel-cepstrum distortion (MCD) and $F_0$ Root Mean Square Error, indicating the generated audio was mathematically the closest to the original human recordings.
  • Subjective: In Mean Opinion Score (MOS) tests, human listeners rated the UPF-Enc system as having the highest similarity to the target speaker's natural voice.

Limitations and Future Directions

A notable challenge identified during the study was a performance gap between genders. The models generally performed better on female voices than male voices. The authors attributed this to a significant gender imbalance in the AISHELL-3 training set (140 females vs. 26 males), highlighting how data bias can impact the modeling of prosodic features.

Future work will focus on expanding the set of intuitive features to further refine speaker identity and improving the "disentanglement" of features. The goal is to ensure that prosodic information and speaker identity vectors remain distinct, preventing the model from receiving redundant or conflicting data during the synthesis process.

Summary updated

BibTeX
@inproceedings{cheng2022speaker,
  author = {Pengyu Cheng and Zhen-Hua Ling},
  title = {{Speaker Adaption with Intuitive Prosodic Features for Statistical Parametric Speech Synthesis}},
  booktitle = {ICDSP},
  year = {2022},
  url = {https://openreview.net/forum?id=5HNW2unAiC},
}
Figure