TL;DR
This paper investigates how the length of surrounding speech (contextual length) influences pause durations in expressive read speech. They propose computational models that can predict these pauses, aiming to improve the naturalness of Text-To-Speech (TTS) systems and aid in prosodic analysis by accounting for this contextual bias.
What Problem Does This Paper Address?
Current Text-To-Speech (TTS) systems often produce monotonous speech, lacking the natural prosodic variations found in human speech. While pauses are crucial for expressiveness, accurately modeling their duration, especially in expressive speech, remains a challenge. Previous research has shown that the length of the surrounding speech (contextual length) affects pause durations in neutral speech. This paper aims to extend these findings to expressive speech, specifically focusing on read tales, and to develop models that can be integrated into TTS systems to generate more natural-sounding speech. Furthermore, understanding this contextual influence is vital for accurately analyzing the expressive functions of pauses themselves, as unmodeled contextual effects can bias interpretations.
Methods
The authors employed two main modeling approaches:
- Kernel Regression (Nadaraya-Watson): This non-parametric method was used to directly model the relationship between contextual length and pause duration. They used a bidimensional vector representing the length of the preceding and following speech units (either sentences or pseudo-clauses) as input. The kernel function allowed for weighting observations based on their proximity to the prediction point.
- Parametric Models: Based on observations from the kernel regression, the authors developed 16 parametric models. These models combined basic functions like exponentiation, logarithm, addition, and multiplication to capture the relationship between contextual length and pause duration. They used a non-linear least squares regression (Matlab's
nlinfit) to infer model parameters and employed 20-fold cross-validation for benchmarking.
For both methods, the "contextual length" was primarily measured using the number of phonemes in the preceding and following speech units. Pauses shorter than 40ms were excluded, and pauses not associated with punctuation were assigned a default duration of 40ms to account for pause deletion phenomena.
Datasets
The study utilized the GV-LEx read tale corpus. This corpus consists of twelve tales read by a single professional speaker in studio conditions. The recordings were chosen to exhibit large prosodic variations while minimizing physiological and cognitive influences. The corpus was phonetically transcribed and phoneme-aligned.
Key Insights and Ideas
- Phoneme as the Best Unit: The number of phonemes in surrounding speech units proved to be the most effective measure for contextual length in their models.
- Hierarchical Pause Durations: Pause distributions, when analyzed across different boundary types (within pseudo-clause, pseudo-clause boundary, sentence boundary, paragraph boundary), showed a clear hierarchical tendency: PCW < PCB < SB < PB.
- Preceding vs. Following Context:
- Inter-sentence pauses were more influenced by the length of the preceding sentence.
- Intra-sentence pauses (at pseudo-clause boundaries) were more influenced by the length of the following pseudo-clause. This finding regarding the following context for intra-sentence pauses was noted as novel.
- Expressive Speech Nuances: The study highlights that expressive speech exhibits a higher degree of pausing irregularities, with a significant portion of intra-sentence pauses not being associated with punctuation. This suggests that speakers use pauses expressively beyond grammatical cues.
Key Results
- Kernel Regression Predictions:
- Inter-sentence pauses were predicted to be between 427 and 912 ms at sentence boundaries.
- Intra-sentence pauses were predicted to be between 93 and 347 ms at pseudo-clause boundaries.
- Parametric Model Performance: While no single parametric model significantly outperformed others based on mean squared error, the simplest linear models provided interpretable insights.
- The best linear model for inter-sentence pauses (SB) showed a greater importance for the preceding sentence length.
- The best linear models for intra-sentence pauses (PCB and PCB*) showed a greater importance for the following pseudo-clause length.
- Pause Realization Rate: The likelihood of a pause occurring at a pseudo-clause boundary was strongly dependent on the length of the adjacent pseudo-clauses, increasing significantly when both were longer.
Limitations and Future Work
- Single Speaker Constraint: The models were developed based on data from a single speaker. The authors acknowledge that the validity of the models is constrained to similar speech rates and suggest future empirical studies to map these observations to different speech rates.
- Pause Deletion Complexity: The study attempted to account for pause deletion by assigning a default duration, but the authors note that the lower fit for intra-sentence pause models is likely due to the high number of pauses not realized despite punctuation. Future work should aim to separate the prediction of pause realization from pause duration.
- Exploring Other Descriptors: The authors suggest investigating other potential pause duration predictors, such as syntactic complexity, intonational phrase length, and prosodic structure complexity, especially those that can be inferred automatically for TTS applications.
- Perceptual Validation: The authors recommend perceptual experiments to validate the proposed parametric and non-parametric models.
Summary updated