Evaluating prosodic processing for incremental speech synthesis

Timo Baumann, David SchlangenInterspeech 2012

TL;DR

This paper investigates how much information about an upcoming utterance (lookahead) is needed for incremental speech synthesis to produce natural-sounding prosody. They found that even with a small lookahead, comparable to a single phrase, prosodic quality can be significantly improved, making incremental systems more practical for real-time dialogue applications.

What Problem Does This Paper Address?

Modern speech synthesis systems typically require the entire text of an utterance before they can begin generating speech. This "batch processing" approach is a bottleneck for interactive systems like spoken dialogue agents, which need to respond quickly and adapt to user input in real-time. Incremental speech synthesis (ISS) aims to solve this by producing speech in chunks as input becomes available. However, a key challenge for ISS is that it cannot perform global optimization on the entire utterance's prosody (intonation and rhythm) because it doesn't have access to the full text. This can lead to unnatural-sounding speech. The authors address the question of how much "lookahead" – knowledge of the upcoming text – is necessary to achieve good prosodic quality in an incremental system, balancing responsiveness with naturalness.

Methods

The researchers extended an existing incremental speech synthesis (ISS) component built on the INPROTK architecture. Their core innovation lies in evaluating the impact of varying "lookahead" on the prosodic quality of the synthesized speech. They integrated the non-incremental symbolic pre-processing steps of the MaryTTS system into their ISS framework.

The experiment involved synthesizing utterances under different lookahead conditions, ranging from "trivial" (no lookahead, each phrase synthesized in isolation) to "full phrase lookahead" (the next phrase is requested before the current one begins). They also explored intermediate lookahead settings, denoted by w1, w2, etc., where w1 means one word of lookahead, and wn-1 means one word before the end of the current phrase. A crucial aspect is that they used "left context" (information from previously spoken words) in most of their incremental conditions, which helps in connecting prosodic phrases smoothly. They compared the prosody (pitch and duration) of the incrementally synthesized speech against a non-incremental "gold standard" using Root Mean Squared Error (RMSE).

Datasets

The authors generated six complex utterances from a German calendar domain. These utterances were designed to simulate typical interactions in a spoken dialogue system, containing 6-7 "chunks" (units produced by an incremental Natural Language Generation component) and averaging 27 words per utterance. They synthesized approximately 70 seconds of speech for each experimental condition.

Key Insights and Ideas

  • Lookahead is Crucial for Prosody: The paper strongly demonstrates that even a limited amount of lookahead significantly improves prosodic quality compared to synthesizing phrases in complete isolation.
  • The "Sweet Spot" for Lookahead: They found that a lookahead of roughly one phrase (equivalent to their w1 condition) offers a good balance between timely system reaction and natural prosody. This level of lookahead allows the system to adapt the prosody of the current phrase based on its continuation, leading to smoother and more natural intonation.
  • Left Context is a Powerful Aid: Even without any lookahead, using left context (information from previously spoken words) drastically reduces prosodic deviations. This highlights the importance of maintaining continuity in speech.
  • Incrementalization of Existing Systems Works: The approach of wrapping existing, non-incremental prosody processors within an incremental framework proved surprisingly effective, suggesting that fully integrated incremental prosody processing might yield even better results.

Key Results

The study quantified the deviation from a non-incremental gold standard using RMSE for both timing and pitch.

  • Full Phrase Lookahead (w0): Achieved the best results, with timing deviation of 0.81 ms (1% of mean phoneme duration) and pitch deviation of 7.08 Hz (4.1% of mean pitch). These values are close to just-noticeable differences for speech tempo and pitch.
  • One Phrase Lookahead (w1): Showed a slight increase in deviation (1.16 ms for timing, 8.32 Hz for pitch) but still performed very well.
  • No Lookahead with Left Context (Wn): Performed significantly worse than conditions with lookahead, indicating that while left context helps, it's not a complete substitute for knowing what's coming next.
  • Trivial Incremental (Wn): The worst performer, with substantial deviations in both timing and pitch, highlighting the unnaturalness of synthesizing phrases in complete isolation.

The results showed a clear, almost linear relationship between the amount of lookahead and the improvement in prosodic quality.

Limitations and Future Work

The authors acknowledge that their current system uses fixed lookahead and update granularities (full phrases). They suggest that future work could explore:

  • Variable Update Timings: Allowing the system to adjust when it requests new chunks based on context or processing load.
  • Fully Integrated Incremental Prosody: Developing prosody processing modules that are inherently incremental, rather than wrapping existing ones.
  • Handling Errors Gracefully: Investigating how to manage and integrate errors that might arise from late changes in the utterance.
  • Subjective Evaluation: While they performed objective analysis, they plan to conduct subjective user ratings to confirm that these objective improvements translate to perceived naturalness.

Summary updated

BibTeX
@inproceedings{baumann2012evaluating,
  author = {Timo Baumann and David Schlangen},
  title = {{Evaluating prosodic processing for incremental speech synthesis}},
  booktitle = {Interspeech},
  year = {2012},
}

Loading related content...

Figure