Quality analysis of macroprosodic F0 dynamics in text-to-speech signals

Christoph R. Norrenbrock, Florian Hinterleitner, Ulrich Heute, Sebastian MöllerInterspeech 2012

TL;DR

This paper investigates how the naturalness of Text-to-Speech (TTS) systems is influenced by prosodic features, specifically fundamental frequency (F0) dynamics. The authors propose novel instrumental metrics to quantify these dynamics and demonstrate that they can predict perceived speech quality with high accuracy, suggesting that F0 variation is a critical factor in achieving natural-sounding synthetic speech.

What Problem Does This Paper Address?

Generating natural-sounding speech from text (Text-to-Speech, TTS) is a long-standing challenge. While significant progress has been made in reducing spectral artifacts, the prosody – the rhythm, stress, and intonation of speech – remains a key differentiator between synthetic and human speech. Specifically, the paper focuses on macroprosody, which refers to the broader intonational patterns over longer utterances, and how the dynamics of fundamental frequency (F0), a primary acoustic correlate of pitch, contribute to the perceived naturalness of TTS. Existing instrumental quality assessment methods often focus on local artifacts, neglecting these larger-scale prosodic variations. The authors aim to bridge this gap by developing methods to quantify F0 dynamics and link them directly to subjective quality ratings.

Methods

The researchers introduce a set of novel instrumental features designed to capture F0 dynamics. These include:

  • Peakedness Ratio (PR): Measures the proportion of speech segments where the F0 slope (derived from linear regression) exceeds a certain threshold, indicating sharp pitch changes.
  • Drop Ratio (DR): Quantifies the fraction of speech segments with a declining F0 slope.
  • Variability Ratio (VR): Assesses the proportion of segments with a mean derivative (rate of F0 change) above a threshold, reflecting overall pitch variability.

In addition to these, they also consider standard F0 statistics like range, standard deviation, and mean. To model perceptual quality, they employ:

  • Regression Models: Linear regression models are trained to predict subjective quality ratings using the extracted prosodic features.
  • Sequential Feature Selection with Cross-Validation: This technique iteratively selects the most informative features to build the best predictive model, ensuring robustness and generalizability.
  • Reference Mapping: They explore mapping TTS features to those of natural speech to account for the "naturalness gap."
  • Feature Expansion: Nonlinear combinations of features are explored to capture more complex relationships.

Datasets

The study utilizes two main speech signal databases:

  • TTS Database: A collection of 10 German sentences synthesized by 14 male and 15 female state-of-the-art TTS systems (concatenation and HMM-based). This database was used for extensive auditory tests where 30 listeners rated the speech on 16 attribute scales, which were then factor-analyzed into three quality dimensions (naturalness, disturbances, temporal distortions).
  • Natural Speech Database: The same 10 German sentences were recorded from 4 male and 4 female amateur speakers to serve as a reference for natural speech prosody.

Key Insights and Ideas

  • F0 Dynamics are Crucial for Naturalness: The study strongly suggests that the way F0 changes over time (its dynamics) is a primary driver of perceived naturalness in TTS.
  • TTS F0 Dynamics Lag Behind Natural Speech: A significant finding is that TTS systems, across the board, exhibit considerably less F0 variation compared to natural human speech. This "prosodic flatness" is a major contributor to the synthetic sound.
  • Instrumental Metrics Can Predict Quality: The proposed instrumental features, particularly those capturing F0 dynamics, can be used to build robust linear models that predict subjective quality ratings with impressive accuracy.
  • Naturalness is a Foremost Prosodic Dimension: The analysis of the quality space indicates that naturalness is heavily influenced by suprasegmental attributes like accentuation and rhythm, which are closely tied to F0 dynamics.
  • No Need for Natural Reference (for this task): Interestingly, the authors found that mapping TTS features to natural speech references didn't improve prediction accuracy. This suggests that focusing on the differences within TTS signals themselves is more effective for quality prediction than trying to match an external natural speech benchmark, especially given the large gap.

Key Results

  • The study achieved cross-validated correlations of up to 0.83 between the predicted and actual subjective naturalness ratings using raw prosodic features. This explains nearly 70% of the rating variance.
  • When comparing TTS to natural speech, nearly all prosodic parameters showed a remarkable difference in their mean values, with TTS exhibiting significantly lower F0 dynamics.
  • The proposed instrumental features, particularly PR, DR, and VR, showed significant correlations with the "naturalness" factor in the quality space.
  • Linear models built on these features demonstrated excellent predictive performance, with a mean correlation of 0.79 for male TTS and 0.81 for female TTS using the expanded feature set (h1-3(X)).

Limitations and Future Work

  • Focus on F0: The current study primarily analyzes F0 dynamics. While crucial, other prosodic aspects like duration and intensity also contribute to naturalness and were not as deeply investigated.
  • Gender Differences: While the models performed well for both genders, some differences in feature behavior and prediction stability were observed, particularly for male TTS.
  • Future Work: The authors suggest exploring robust acoustical parameters that capture additional quality factors like voice quality and articulation. They also aim to implement a full objective account of TTS quality by integrating these various factors. Further research could also investigate the impact of nonlinear feature combinations more thoroughly.

Summary updated

BibTeX
@inproceedings{norrenbrock2012quality,
  author = {Christoph R. Norrenbrock and Florian Hinterleitner and Ulrich Heute and Sebastian Möller},
  title = {{Quality analysis of macroprosodic F0 dynamics in text-to-speech signals}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure