Towards intonation control in unit selection speech synthesis

Cédric Boidin, Olivier Boeffard, Thierry Moudenc, Géraldine DamnatiInterspeech 2009

TL;DR

This paper tackles the challenge of controlling intonation in unit selection speech synthesis by proposing a novel approach that combines a statistical intonation model (CART-HMM) with the unit selection framework using Finite State Machines (FSMs). While subjective evaluations didn't show a significant improvement in perceived quality, the synthesized intonation was demonstrably closer to natural speech, suggesting the model's effectiveness in capturing prosodic nuances.

What Problem Does This Paper Address?

Traditional unit selection speech synthesis, while effective for neutral speech, struggles with expressive corpora due to the inherent variability in prosody. Prosody is influenced by complex linguistic and para-linguistic factors that are difficult to explicitly model. Existing unit selection methods often rely on acoustic and symbolic prosodic tags but lack a robust mechanism to control the fine-grained numerical aspects of intonation, leading to less natural-sounding expressive speech. This paper aims to bridge this gap by integrating a data-driven statistical intonation model directly into the unit selection process.

Methods

The core of the proposed method is the integration of a mixed CART-HMM intonation model into the unit selection framework using Finite State Machines (FSMs).

  • CART-HMM Intonation Model: This model first stylizes and annotates syllables with linguistic features and acoustic F0 points. A Classification and Regression Tree (CART) classifies intonation patterns based on linguistic information, and a Hidden Markov Model (HMM) captures unexplained intonation variability by introducing hidden states. This allows for multiple prosodic realizations for each linguistic class.
  • Finite State Machine (FSM) Framework: To integrate this intonation model with standard unit selection, the authors employ an FSM framework. They construct three automata:
  • Segmental Automaton (S): Represents the standard diphone-based unit selection, incorporating target and concatenation costs.
  • Transducer (V): Converts acoustic units (diphones) into vocal nuclei, bridging the gap between segmental units and the syllable-based intonation model.
  • Intonation Automaton (I): Encodes the HMM state lattice produced by the intonation model, measuring the quality of intonation for sequences of vocal nuclei.
  • Joint Automaton (J): These three automata are composed (J = S * V * I) to create a joint automaton that incorporates both segmental and prosodic constraints. The best path through this joint automaton yields the selected units for synthesized speech.
  • Pruning: To manage computational complexity, the segmental automaton (S) is pruned before composition, significantly reducing its size while aiming to preserve vocal nuclei.

Datasets

The experiments were conducted on four French speech corpora designed for the Orange Labs unit selection speech synthesizer. These corpora were recorded by four professional speakers (Agnes, Lise, Julie, Loic). One speaker (Agnes) was constrained to a neutral style, while the others provided more expressive speech. The corpora served as both the acoustic inventory for unit selection and a subset was used for training the intonation model. No new benchmarks were introduced; existing corpora were utilized.

Key Insights and Ideas

  • FSMs as a Unifying Framework: The use of FSMs proves to be a flexible and powerful way to combine different types of models (segmental unit selection and statistical intonation) with varying unit granularities (diphones, vocal nuclei, HMM states).
  • Bridging Segmental and Prosodic Control: The transducer V plays a crucial role in harmonizing the diphone-based segmental units with the syllable-based intonation model by focusing on vocal nuclei.
  • Intonation Model Captures Nuance: Even if not leading to statistically significant subjective improvements, the F0 features of the JOINT synthesized speech were closer to the corpus intrinsic values than the SEG version, indicating the intonation model successfully influenced the prosody.
  • Trade-offs in Joint Optimization: The authors acknowledge that the joint selection might involve a compromise between segmental and intonation constraints, potentially leading to a slight loss in segmental quality to achieve better prosody.

Key Results

  • Subjective Evaluation: While the "natural" speech was rated highest, the JOINT (segmental + intonation) and SEG (segmental only) versions did not show a statistically significant difference in subjective ratings across the four voices or overall. The JOINT version was rated slightly better than SEG in overall mean rating (0.23 difference) and for two speakers (Lise and Julie).
  • Objective Acoustic Analysis: Table 3 shows that the F0 mean and standard deviation of the JOINT synthesized scenarios were globally closer to the corpus intrinsic values compared to the SEG scenarios. This confirms the beneficial influence of the intonation model on the acoustic realization of prosody.

Limitations and Future Work

  • Limitations:
    • The subjective evaluations did not yield statistically significant improvements for the JOINT version over the SEG version.
  • The joint selection process might involve a compromise between segmental and intonation constraints.
  • Other prosodic factors, such as phone duration and break positions, were not included in this study and could contribute to further improvements.
  • Future Work: The authors suggest including other prosodic factors, like phone duration, into the intonation model to achieve a more comprehensive control of expressive speech synthesis.

Summary updated

BibTeX
@inproceedings{boidin2009towards,
  author = {Cédric Boidin and Olivier Boeffard and Thierry Moudenc and Géraldine Damnati},
  title = {{Towards intonation control in unit selection speech synthesis}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure