TL;DR
This paper tackles the challenge of creating natural-sounding intonation in speech synthesis by modeling the "f0 reset" – the pitch adjustment that occurs after a pause within a sentence. They propose and evaluate both linear regression and neural network models to predict this pitch reset, demonstrating that these models can significantly improve the tonal coherence of synthesized speech.
What Problem Does This Paper Address?
Speech synthesis systems that rely on concatenating pre-recorded speech units (corpus-based approaches) often struggle with natural intonation, particularly when sentences contain internal pauses. While these systems can generate natural-sounding individual "phonic groups" (segments of speech between pauses), stitching them together can result in a jarring lack of tonal coherence. This is because the fundamental frequency (f0) at the beginning of a new phonic group after a pause doesn't always smoothly follow the end of the previous one. The paper specifically addresses the "f0 reset" phenomenon – how the pitch adjusts after an internal pause to maintain a natural flow and tonal connection across sentence segments. Existing corpus-based methods often treat internal pauses similarly to sentence beginnings or endings, failing to capture the nuanced pitch adjustments required for naturalness.
Methods
The authors propose two main approaches to model the f0 reset:
- Linear Regression: They employ a stepwise linear regression model. This method starts with a set of potential input features and incrementally adds the most statistically significant ones to predict the f0 at the beginning of the next phonic group. The model aims to find a linear combination of features that best explains the f0 reset.
- Neural Network: They utilize a multi-layer perceptron (MLP) with a single hidden layer. This approach is more flexible than linear regression and can capture non-linear relationships between input features and the target f0 reset. They explore various network configurations and input parameter combinations to find the optimal model.
Both methods are trained automatically on a corpus, avoiding the need for expert linguistic knowledge.
Datasets
The study uses a corpus of approximately 1300 isolated sentences recorded by two male speakers. This corpus was enriched with:
- 800 manually designed sentences to cover a rich prosodic sample of the Galician language.
- 500 automatically collected sentences to include more complex prosodic structures.
The fundamental frequency (f0) contours were extracted using Praat, with post-processing to remove voiceless regions. The corpus was organized into "accent groups" (sequences of non-accented words ending in an accented word) and "phonic groups" (sequences of words between pauses). Accent groups were further clustered into 48 classes based on proposition type, position within the phonic group, and accent position within the accent group.
Key Insights and Ideas
- The f0 reset is crucial for tonal coherence: The paper strongly demonstrates that the pitch at the beginning of a phonic group after an internal pause is significantly different from the pitch at the end of the preceding group. This difference is not random and needs to be modeled to avoid unnatural-sounding synthesized speech.
- Sentence-internal pauses require specific modeling: Treating internal pauses the same as sentence beginnings or endings is insufficient. The pitch behavior after an internal pause has its own distinct characteristics.
- Multiple factors influence the f0 reset: The study identifies several features that influence the f0 reset, including the number of accent groups in the next phonic group, the position of the phonic group within the sentence, and crucially, the duration of the pause itself.
- Neural networks offer an advantage: While linear regression provides a good baseline, the neural network model, despite being more complex to train, ultimately achieves better performance, particularly in terms of error variance, suggesting it can better capture the subtle complexities of f0 reset.
- The final f0 of the preceding group is a strong predictor: Both models find that the f0 value at the end of the preceding phonic group is a highly significant predictor of the f0 reset.
Key Results
- Significant f0 differences: Histograms show clear differences in f0 values for phonic groups that are sentence-initial versus those that are not, and similarly for phonic groups at the end of a sentence versus those that are not. For example, sentence-initial phonic groups have an average f0 of 101.96 Hz, while non-sentence-initial ones average 81.88 Hz.
- Linear Regression Model: The linear regression model identified several significant predictors for the f0 reset, with the final f0 of the preceding group (Freqi-1) being the most important. Other significant factors included pause duration (Dur), number of accent groups in the next phonic group (Ag2), and phonic group position (PgPos).
- Neural Network Model: The best-performing neural network model used four input parameters: phonic group position (PgPos), number of accent groups in the next phonic group (Ag2), pause duration (Dur), and the final frequency before the pause (Freqi-1). This model achieved a lower Root Mean Square Error (RMSE) of 7.92 Hz and a higher explained variance (R2) of 0.74 compared to the linear regression model (RMSE 8.59 Hz, R2 0.70).
- Qualitative Improvement: The models were integrated into a corpus-based speech synthesizer, and preliminary informal listening tests showed encouraging results, improving the tonal coherence of synthesized speech.
Limitations and Future Work
- Linguistic Classification: The authors suggest that a finer linguistic classification of sentence types, particularly for complex structures like enumerations or explanations, could further improve the linear regression model.
- Generalizability: While the models were integrated into a bilingual Galician and Spanish synthesizer, a more thorough evaluation is needed to confirm their effectiveness across different languages and speaker characteristics.
- Superpositional Models: The authors note that the proposed f0 reset modeling approach could be applied to other intonation modeling paradigms, such as superpositional models, not just concatenative ones.
- Thresholding: For practical implementation in unit selection, a threshold was introduced to allow for a small degree of variability in the f0 reset, giving a null concatenation cost to accent groups with initial f0 values close to the predicted value. This threshold (10 Hz) was found to be reasonable in informal tests.
Summary updated