Text-to-speech intelligibility across speech rates

Ann K. Syrdal, H. Timothy Bunnell, Susan R. Hertz, Taniya Mishra, Murray Spiegel, Corine Bickley, Deborah Rekart, Matthew J. MakashayInterspeech 2012

TL;DR

This paper presents a comprehensive study evaluating the intelligibility of eight different text-to-speech (TTS) systems across a range of speech rates, from 200 to 450 words per minute. The findings reveal significant differences in how well these systems maintain intelligibility as speech speeds up, with unit selection systems generally outperforming others, and one even achieving parity with human speech.

What Problem Does This Paper Address?

As text-to-speech (TTS) technology advances, a critical factor for user adoption and satisfaction is intelligibility, especially at faster speech rates. Users often desire quicker access to information, but increasing speech speed can degrade comprehension. This paper addresses the challenge of systematically comparing the intelligibility of various TTS synthesis methods under different speaking speeds. The authors aim to provide benchmark data and understand how factors like synthesis technique, speech rate, and listener characteristics influence comprehension, thereby informing the development of more robust and user-friendly TTS systems.

Methods

The study employed a web-based, open-response listening test to measure TTS intelligibility. Participants listened to semantically unpredictable sentences (SUSs) generated by eight different TTS systems and a linearly time-compressed human speech reference. These TTS systems represented four distinct synthesis methods: formant, diphone concatenation, unit selection concatenation, and Hidden Markov Model (HMM) synthesis, with two independent systems for each method. For each TTS system, both a female and a male American English voice were tested (with one exception).

Sentences were presented at six speech rates: 200, 250, 300, 350, 400, and 450 words per minute (wpm). Listeners were instructed to type exactly what they heard, with provisions for indicating unclear or unfamiliar words. Transcriptions were automatically scored using a normalized edit distance (NED) metric, which essentially represents the proportion of words misheard. A mixed-effects ANOVA was used to analyze the data, considering speech rate as a within-subject factor and speech source, voice gender, audio equipment, listener age, and native language as between-subject factors.

Datasets

The study utilized a set of 6-to-8-word semantically unpredictable sentences (SUSs) generated by the SUSgen program. These sentences were designed to be syntactically grammatical but semantically anomalous, ensuring a diverse range of vocabulary and phonetic content. The vocabulary included familiar monosyllabic and polysyllabic words, a wide variety of phonemes, consonant clusters, stressed and unstressed syllables, and content and function words. The paper does not describe the creation of a new benchmark dataset but rather uses existing methods for generating test materials and evaluating intelligibility.

Key Insights and Ideas

  • Synthesis Method Matters, But Specifics are Key: While unit selection systems generally performed best, the study suggests that specific implementations within a synthesis method are more critical than the method itself. There isn't a single "best" synthesis technology across all conditions.
  • Human Speech as a Benchmark: The linearly time-compressed human speech reference served as a crucial benchmark, with one unit selection TTS system achieving statistically equivalent intelligibility. This highlights the potential for TTS to reach human-like performance.
  • Speech Rate is a Universal Challenge: As expected, intelligibility consistently decreased as speech rate increased across all systems. However, the rate of degradation varied significantly between systems.
  • Listener Factors Influence Comprehension: Beyond the TTS system, listener characteristics like native language, familiarity with TTS, and audio equipment (headphones vs. speakers) played a significant role in intelligibility scores.
  • Age is Less of a Factor Than Expected: Surprisingly, listener age did not show a significant main effect on intelligibility, possibly due to older listeners having greater familiarity with TTS technology.

Key Results

  • Unit Selection Dominance: The two unit selection TTS systems were the most intelligible across speech rates, with one system (U2) performing as well as or better than the human speech reference.
  • Speech Rate Impact: Errors increased significantly with every 50 wpm rise in speech rate.
  • System Variability: While most systems showed a gradual increase in errors with speed, some degraded more rapidly (e.g., H1), while others showed a smaller increase (e.g., F2).
  • Audio Equipment Advantage: Using headphones resulted in 6.2% fewer errors compared to speakers.
  • Native Language Benefit: Native English speakers made 9.1% fewer errors than non-native speakers.
  • TTS Familiarity: Listeners who reported being more familiar with TTS generally had lower error rates.

Limitations and Future Work

The authors acknowledge that the means by which each TTS system controls its speech rate are proprietary and not revealed. They also note that one diphone concatenation system (C2) could only be tested at lower rates due to constraints. The study's findings suggest that the proposed methodology is sensitive enough to detect differences among TTS systems and speech manipulations. Future work could explore the impact of different types of speech material (e.g., spontaneous vs. read speech) and investigate the specific acoustic features that contribute to intelligibility at faster rates. Further research into the interaction between listener characteristics and TTS performance could also lead to more personalized and effective TTS systems.

Summary updated

BibTeX
@inproceedings{syrdal2012text,
  author = {Ann K. Syrdal and H. Timothy Bunnell and Susan R. Hertz and Taniya Mishra and Murray Spiegel and Corine Bickley and Deborah Rekart and Matthew J. Makashay},
  title = {{Text-to-speech intelligibility across speech rates}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure