Synthesis of fast speech with interpolation of adapted HSMMs and its evaluation by blind and sighted listeners

Michael Pucher, Dietmar Schabus, Junichi YamagishiInterspeech 2010

TL;DR

This paper tackles the challenge of generating highly intelligible synthetic speech at very fast speaking rates, a crucial feature for visually impaired users. They introduce a novel method using interpolation of adapted Hidden Semi-Markov Models (HSMMs) to smoothly transition between normal and fast speech characteristics, significantly improving both intelligibility and voice quality compared to standard techniques.

What Problem Does This Paper Address?

Synthetic speech is a vital tool for visually impaired individuals, enabling them to access information more efficiently. To maximize the amount of information conveyed, faster speaking rates are highly desirable. However, existing data-driven speech synthesis methods often suffer a significant drop in intelligibility and voice quality when pushed to very high speaking rates. This paper aims to bridge this gap by developing a method that can generate natural-sounding and understandable speech even at speeds exceeding typical human conversational rates.

Methods

The core of the proposed solution lies in adapting and interpolating Hidden Semi-Markov Models (HSMMs), a standard framework for speech synthesis. The authors first train HSMMs on speech data recorded at both normal and fast speaking rates for specific speakers. They then adapt the normal-rate models to create fast-rate models. The key innovation is the interpolation technique:

  • HSMM Interpolation: Instead of using a single model for fast speech, they create a continuous spectrum of models by interpolating between the normal and fast-rate HSMMs. This allows for fine-grained control over the speaking rate.
  • Feature Interpolation: The interpolation can be applied to different aspects of the HSMMs, including state duration probabilities, spectral features, and fundamental frequency (F0). The paper explores two variants: interpolating only duration (SPO-SPO+F) and interpolating duration, spectrum, and F0 (SPO-SPF).
  • Context-Dependent Adaptation: They use context-dependent linear regression functions for HSMM duration adaptation, incorporating linguistic features like syllable and phrase information to better model complex duration changes.

Datasets

The evaluation was performed using speech data from two Austrian German male speakers (SPO and HPO). The data consisted of phonetically balanced corpora of approximately 300 sentences recorded at both normal and fast speaking rates for each speaker. No new datasets were introduced; existing speaker-specific corpora were utilized.

Key Insights and Ideas

  • Blind Listeners Excel at Fast Speech: The study revealed that blind listeners, who are accustomed to processing information at higher rates, demonstrated significantly better comprehension of fast synthetic speech compared to sighted listeners. This highlights the critical importance of high-quality fast speech synthesis for this user group.
  • Interpolation is Key to Naturalness: Simply adapting models for fast speech isn't enough. The interpolation technique is crucial for achieving better non-linear modeling of duration, leading to more natural-sounding speech, especially in challenging linguistic contexts where standard methods struggle.
  • Duration Modeling is a Bottleneck: Analysis of standard HSMM duration modeling for fast speech identified specific linguistic contexts that are particularly problematic, suggesting areas for future research and model improvement.
  • SPO-SPO+F Outperforms: The method interpolating only duration (SPO-SPO+F) showed a significant advantage over standard methods and even the method interpolating all features (SPO-SPF) in terms of listener preference and lower word error rates. This suggests that duration is the most critical factor for intelligibility at high speeds.

Key Results

  • Improved Intelligibility: The proposed interpolation method (SPO-SPO+F) achieved lower word error rates (WER) compared to standard HSMM-based duration modeling. For blind listeners, the WER was 16.4% compared to 23.2% for the standard method.
  • Enhanced Voice Quality: In pairwise comparison tests, listeners (both blind and sighted) significantly preferred sentences generated with the interpolation methods (SPO-SPO+F and HPO-HPO+F) over standard methods.
  • High Speaking Rates Achieved: The fastest sentences generated contained up to 9 syllables per second, with blind listeners showing particularly good comprehension at rates of 8-9 syllables per second.

Limitations and Future Work

  • Limited Adaptation Data: The proposed method can be used with limited amounts of fast speech adaptation data, which is a strength. However, the authors acknowledge that further investigation into the optimal amount of data is needed.
  • Investigating User Habits: Future work will focus on understanding the actual duration rates used by blind users in their daily interactions with speech synthesis.
  • Deeper Error Analysis: The authors plan to conduct a more detailed analysis of duration modeling errors, going beyond comparing complete models to examining specific linguistic contexts.
  • Cross-Speaker Adaptation: Investigating the use of fast speech background models and fast duration models from one speaker for another speaker is also a future direction.

Summary updated

BibTeX
@inproceedings{pucher2010synthesis,
  author = {Michael Pucher and Dietmar Schabus and Junichi Yamagishi},
  title = {{Synthesis of fast speech with interpolation of adapted HSMMs and its evaluation by blind and sighted listeners}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure