TL;DR
This paper introduces a novel method for adapting speech prosody (specifically, the fundamental frequency or F0 contour) to new speakers or speaking styles. By focusing on modifying the commands that control an F0 generation model, rather than directly manipulating F0 values, the approach achieves more natural and flexible prosody adaptation, even with limited training data.
What Problem Does This Paper Address?
Speech synthesis systems, particularly those based on Hidden Markov Models (HMMs), have made significant strides in generating natural-sounding speech. However, controlling prosody – the rhythm, stress, and intonation of speech – remains a challenge. Traditional HMM-based methods often lead to over-smoothed or unnaturally abrupt F0 contours, especially when training data is scarce. Furthermore, adapting these systems to new speakers or distinct speaking styles (like polite vs. impolite speech) often relies on simple linear transformations of F0, which fail to capture the complex prosodic structures and can result in unnatural-sounding speech. This paper addresses the need for a more robust and flexible method for prosody adaptation and voice conversion that can handle significant changes in F0 contours while maintaining speech quality.
Methods
The core of the proposed method lies in leveraging the Fujisaki generation process model for F0 contours. This model represents F0 as a combination of phrase and accent components, controlled by specific "commands" (magnitudes and timings). Instead of directly adapting the F0 values frame-by-frame, the authors propose to:
- Predict Differences in Commands: They extract the F0 model commands for both original and target speech utterances. Then, they train binary decision trees (BDTs) to predict the differences in the magnitudes/amplitudes of these commands between the original and target styles/speakers. This is a key innovation, as it focuses on the underlying control signals rather than the output F0.
- Apply Differences: These predicted command differences are then applied to the original F0 model commands. Phrase command magnitudes are adjusted first, followed by accent command amplitudes, taking the modified phrase commands into account.
This approach allows for adaptation even when the training corpus for command differences is small, and importantly, the speakers used for training command differences do not need to be the same as the speakers in the baseline system. The method also incorporates adaptation of pause and phone durations.
Datasets
- ATR Continuous Speech Corpus: Used for experiments. This corpus contains 503 sentences.
- Speaker FTY: A female narrator whose neutral reading style was adapted to "polite" and "impolite" styles.
- Speaker MMI: A male narrator whose utterances were used to train polite style conversion differences, which were then applied to FTY's speech.
- Speaker MHT: A male narrator whose utterances served as original utterances for voice conversion experiments.
- Speaker FKS: A female narrator whose utterances served as target utterances for voice conversion experiments.
Key Insights and Ideas
- Command-Level Adaptation is Key: Adapting the underlying control signals (F0 model commands) is more effective and flexible than directly manipulating F0 values. This allows for more natural prosody changes.
- Focusing on Differences: Training BDTs to predict differences in commands, rather than absolute values, significantly reduces the amount of training data required.
- Speaker Independence: The method allows for training command differences using speakers different from those in the baseline system, increasing its practicality.
- Style vs. Emotion: The paper highlights that while the method is effective for style adaptation (polite/impolite), adapting to highly emotional speech, which involves larger prosodic structure changes, is more challenging and requires further investigation.
Key Results
- Style Adaptation: Listening experiments showed that the adaptation to "polite" speech was effective, with average scores indicating good realization. Adaptation to "impolite" speech was less successful, suggesting that more significant prosodic structural changes might be needed.
- Voice Conversion: Objective evaluation (Root Mean Square Error of log F0) showed a slight improvement with the proposed method (20.73) compared to a baseline linear conversion method (23.05). Subjective listening tests also favored the proposed method, with an average score of 0.419 indicating it was closer to the target speech.
- Naturalness: For polite style adaptation, no apparent degradation in the naturalness of synthetic speech was observed. Degradation was more noticeable for impolite style adaptation.
Limitations and Future Work
- Handling Large Prosodic Structure Changes: The current method struggles with significant prosodic shifts, such as those found in emotional speech. Future work should explore how to model and adapt these larger structural changes.
- Modeling Micro-Prosody: The Fujisaki model does not capture fine-grained F0 undulations (micro-prosody). The authors suggest that these could be modeled as residuals of the Fo model approximation and added to the adapted contours.
- Improving "Impolite" Style Adaptation: The less successful adaptation to "impolite" speech indicates a need to better capture the specific prosodic characteristics of this style.
- Integration with HMM Synthesis: The authors mention a method to reshape HMM-generated F0 contours using the generation process model, suggesting that combining this with their command-based adaptation could lead to even more flexible prosody control.
Summary updated