An empirical text transformation method for spontaneous speech synthesizers

Shiva Sundaram, Shrikanth S. NarayananInterspeech 2003 · DOI

TL;DR

This paper introduces an empirical method to transform written text into a format suitable for spontaneous speech synthesis. By analyzing and incorporating features like filled pauses ("uh," "um") and audible breathing, the system aims to make synthesized speech sound more natural and human-like, bridging the gap between formal text and the nuances of everyday speech.

What Problem Does This Paper Address?

Synthesizing natural-sounding speech is a long-standing goal in human-computer interaction. While traditional text-to-speech (TTS) systems excel at producing clear, formal speech, they often fail to capture the characteristics of spontaneous, conversational speech. Spontaneous speech is rich with features like filled pauses (e.g., "uh," "um"), false starts, laughter, and audible breathing, which are absent in written text but crucial for conveying naturalness and speaker-specific traits. Existing TTS systems struggle to incorporate these features due to the limitations of formal grammars and the difficulty in acquiring large, annotated datasets of spontaneous speech. This paper addresses the challenge of transforming plain text into a representation that can be used by a spontaneous speech synthesizer, focusing specifically on the text analysis stage.

Methods

The core of the proposed method is an empirical text transformation technique that leverages psycholinguistic insights and statistical analysis. The process involves two main stages:

  • Offline Training: This stage involves analyzing a corpus of spontaneous speech to identify patterns preceding filled pauses ("uh" and "um"). This includes:
  • Creating lists of words most likely to precede fillers (Wlistun, Wlistum).
  • Building Finite State Automata (FSA) networks that represent the phrase structures preceding fillers, using part-of-speech (POS) tags for words. These FSAs are then minimized for efficiency.
  • Breathing instances are handled separately, with heuristic rules developed based on observations.
  • Online Processing: When new text is input for synthesis, the system performs:
  • Part-of-speech tagging of the input text.
  • Searching for keywords from the Wlist.
  • Extracting and constructing class-based phrases preceding keywords.
  • Using the trained FSA networks to determine if a filler ("uh" or "um") should be inserted after a keyword.
  • Breathing instances are inserted based on heuristic rules, such as at the beginning of sentences, before conjunctions, or randomly before "um" fillers.

The implementation utilizes the AT&T Finite State Machine Tools and the Edinburgh Language Technology Group's POS tagger.

Datasets

The primary dataset used for offline training and analysis was approximately 190 minutes of transcribed spontaneous lecture monologues. This corpus contained 23,725 "clean" text sentences and 26,618 words annotated with spontaneous speech feature tags. The data was sourced from the University of Southern California (USC) Distance Education Network (DEN) and transcribed at the Speech Analysis and Interpretation Laboratory at USC.

Key Insights and Ideas

  • Fillers as Intentional Features: The paper highlights that filled pauses ("uh," "um") are not mere ungrammatical utterances but intentional linguistic tools used by speakers to manage speech flow, indicate delays, or hold the listener's attention.
  • Speaker-Specific Traits: The usage patterns of fillers and breathing are shown to reflect individual speaker characteristics, making empirical, speaker-adaptive techniques valuable.
  • Class-Based Analysis for Robustness: Using POS tags for phrase analysis (class-based FSA) makes the system more robust to variations in specific words and allows for better generalization across different input texts.
  • Breathing as a Heuristic Rule: Due to the complex and often involuntary nature of breathing, a rule-based heuristic approach is deemed more practical for insertion than purely statistical methods.
  • Empirical Methods for Data Scarcity: The authors emphasize the need for empirical techniques due to the difficulty and expense of collecting large, consistently annotated spontaneous speech datasets, especially for specific speakers or domains.

Key Results

The paper demonstrates the successful implementation of a text transformation system that can insert filled pauses ("uh," "um") and breathing instances into plain text. Examples provided show transformations of sentences like:

  • "I am sorry the number you have dialed does not exist" to "I am sorry [BREATHE IN] the [UH] [PAUSE] number you have dialed does not exist."
  • "Mary had a little lamb but its fleece was not white" to "[BREATHE IN] Mary had a little lamb but its [UM] fleece was not white."

The system was tested on both training and non-training domain sentences, showing effectiveness due to the class-based FSA approach. The authors note that the keyword/keyword class locator helps limit filler occurrences to appropriate places, and by adjusting the Wlist, filler occurrence rates can be varied.

Limitations and Future Work

  • Evaluation Challenges: The authors acknowledge the difficulty in objectively evaluating spontaneous speech synthesis due to the inherent variability even within a single speaker and the lack of a complete, artifact-free spontaneous speech synthesizer for subjective testing.
  • Data Limitations: The reliance on lecture monologues, while convenient, might not fully represent the nuances of conversational speech. The paper notes that conversational corpora like SWITCHBOARD are not ideal for speaker-dependent analysis.
  • Speaker Independence: While the method is adaptable for speaker-independent synthesis, this aspect was not fully explored due to data limitations.
  • Future Work:
    • Developing domain-dependent subjective tests with a complete end-to-end spontaneous speech synthesizer.
  • Exploring methods to capture a wider spectrum of speaker style variations, potentially by working at the signal level.
  • Investigating how these techniques can benefit multi-modal machine interaction and Natural Language Generation for dialogue systems.
  • Addressing the issue of unintentional hesitations caused by simply inserting features, aiming for more seamless integration.

Summary updated

BibTeX
@inproceedings{sundaram2003empirical,
  author = {Shiva Sundaram and Shrikanth S. Narayanan},
  title = {{An empirical text transformation method for spontaneous speech synthesizers}},
  booktitle = {Interspeech},
  year = {2003},
  doi = {10.21437/eurospeech.2003-392},
}
Figure