Off the Cuff: Exploring Extemporaneous Speech Delivery with TTS

Éva Székely, Gustav Eje Henter, Jonas Beskow, Joakim GustafsonInterspeech 2019

TL;DR

This paper introduces a novel platform for exploring extemporaneous speech delivery using a spontaneous conversational speech synthesizer. By generating multiple versions of speech samples with varying fluency and filled-pause placement, the platform allows for a more nuanced, production-oriented evaluation of TTS systems, moving beyond traditional isolated utterance assessments.

What Problem Does This Paper Address?

Public speaking styles can be broadly categorized, with extemporaneous speaking being a common and effective method that blends structure with conversational delivery. Traditional Text-to-Speech (TTS) systems often struggle to replicate this natural, "off-the-cuff" style, typically producing speech that sounds overly scripted or robotic. This is partly because TTS evaluation has historically focused on isolated, read utterances, failing to capture the dynamic and context-dependent nature of spontaneous speech. The authors aim to bridge this gap by developing a system that simulates extemporaneous speech and provides a framework for evaluating TTS from a production perspective, considering how users might interact with and deploy synthesized speech in real-world scenarios.

Methods

The core of this work lies in a spontaneous speech synthesizer built upon the Tacotron 2 framework. The authors leverage a dataset of untranscribed technology podcast recordings ("ThinkComputers" Corpus - TCC) and employ techniques like speaker-dependent breath detection for utterance segmentation. A key innovation is the generation of multiple speech variants by manipulating filled pauses (FPs) and fluency. These variants include:

  • AutoFP: Synthesizes speech with automatically placed FPs, without explicit annotation.
  • CtrlFP: Trained with explicitly annotated FP locations and types ('uh'/'um'), allowing for controlled synthesis with or without FPs.
  • GenFP: Uses a generic FP label, allowing the system to decide the FP type ('uh'/'um') at synthesis time.
  • HalfFluent: Trained on a subset of the data with minimal disfluencies to achieve higher fluency.
  • TransFluent: Utilizes transfer learning from AutoFP, fine-tuned on the more fluent data to balance fluency and naturalness.

The synthesized speech samples are then presented through the Farrago soundboard app, offering an interactive grid interface for playback, color-coding, and customized settings.

Datasets

  • ThinkComputers Corpus (TCC): An untranscribed, public domain weekly technology podcast. This dataset was used for training the speech synthesis models.
  • Interspeech Keynote Speeches: Segments from two keynote speeches were used as prompts for generating the synthetic speech samples demonstrated in the platform.

Key Insights and Ideas

  • Moving Beyond Read Speech: The research highlights the increasing feasibility of simulating spontaneous speech styles with modern TTS, opening doors for more natural and engaging synthetic voices.
  • The Nuance of Filled Pauses: Filled pauses are not just noise; they can influence perceptions of authenticity and engagement, suggesting that their controlled placement is crucial for naturalness.
  • Production-Oriented Evaluation: The paper advocates for evaluating TTS not just on isolated utterances, but on how effectively users can interact with and deploy the synthesized speech to achieve communicative goals.
  • Situational Context Matters: The platform's design emphasizes how different speech variants can be perceived differently depending on the context, suggesting that a one-size-fits-all approach to TTS is insufficient.
  • Potential for AAC: The work points to significant potential for this technology in Augmentative and Alternative Communication (AAC), enabling users to prepare and deliver more natural-sounding speech.

Key Results

The paper doesn't present quantitative results in terms of objective metrics like Mean Opinion Scores (MOS) for the synthesized speech variants. Instead, it focuses on the qualitative aspects and the potential of the platform. The key "result" is the development of the interactive platform itself, which allows users to:

  • Generate and play back six different synthetic speech variants of the same utterance.
  • Color-code samples based on subjective perceptual aspects (e.g., certainty, authenticity).
  • Explore how variations in fluency and filled-pause placement affect the perceived naturalness and expressiveness of extemporaneous speech.

The authors reference prior work [5] suggesting that FPs increase perceived authenticity, while ongoing work is exploring perceptual evaluations of the presented voice variants.

Limitations and Future Work

  • Subjective Evaluation: The current evaluation is primarily subjective and exploratory, relying on user interaction with the Farrago app. Formal perceptual studies are needed to quantify the differences between the voice variants.
  • Limited Data: The TCC corpus, while useful, is untranscribed and consists of only two male speakers. Expanding the dataset with more diverse speakers and transcribed data could improve model robustness.
  • Contextual Understanding: While the platform simulates extemporaneous speech, deeper contextual understanding and adaptation for truly dynamic, real-time interaction remain challenges.

Future work will likely involve conducting more rigorous perceptual studies to validate the effectiveness of the different speech variants and further refine the TTS models for more natural and context-aware spontaneous speech synthesis. The authors also envision expanding the platform's capabilities for AAC users and exploring more sophisticated evaluation methodologies.

Summary updated

BibTeX
@inproceedings{szekely2019off,
  author = {Éva Székely and Gustav Eje Henter and Jonas Beskow and Joakim Gustafson},
  title = {{Off the Cuff: Exploring Extemporaneous Speech Delivery with TTS}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure