DisfluencySpeech - Single-Speaker Conversational Speech Dataset with Paralanguage

Kyra Wang, Dorien Herremans

Abstract

Modern Text-to-Speech (TTS) systems often struggle to replicate the natural nuances of human conversation, such as disfluencies and paralanguage. We introduce DisfluencySpeech, a 10-hour, single-speaker dataset derived from the Switchboard-1 corpus. By providing transcripts at three levels of disfluency, we investigate the ability of TTS models to implicitly learn paralinguistic features. Our results show that while models can learn to generate non-lexical sounds like laughter from semantic cues, significant mismatches between text and audio lead to attention failures in standard Transformer architectures.

Introduction

While current TTS models achieve high fidelity in reading tasks, they often lack the "messiness" of real-world interaction. Human speech is characterized by paralanguage (e.g., sighs, laughter) and disfluencies (e.g., filled pauses like "uh," false starts, and repetitions). These elements are not mere noise; they convey critical social and emotional context. Existing datasets are typically either high-quality but "clean" (e.g., LJSpeech) or conversational but multi-speaker and fragmented (e.g., Switchboard). DisfluencySpeech bridges this gap by providing a high-quality, single-speaker conversational dataset designed to help models master these nuances.

Methods

The dataset consists of 5,000 studio-quality clips recorded by a professional 28-year-old Singaporean female speaker. The scripts were based on expressive utterances from the Switchboard-1 Telephone Speech Corpus. To evaluate how models handle varying degrees of textual information, the researchers developed three transcript levels:

  • Transcript A: Includes all spoken words, filled pauses, and restarts, but excludes non-speech events (laughter/sighs) from the text.
  • Transcript B: Removes filled pauses and discourse markers (e.g., "you know").
  • Transcript C: The cleanest version, further removing false starts.
Comparison of the three transcript levels (A, B, and C) used for training.

Benchmark Transformer-based TTS models were trained on these levels to assess their capacity for implicit learning of missing paralinguistic features.

Key Results

The study utilized Mean Cepstral Distortion (MCD) and Character Error Rate (CER) to evaluate performance across the different transcript levels.

Performance metrics (MCD and CER) for benchmark models across different datasets and transcript levels.

Audio Quality and Phonetic Similarity

The original DisfluencySpeech audio achieved a CER of 4.04%, comparable to the 3.30% of the gold-standard LJSpeech dataset, confirming its studio-quality status. Notably, the model trained on Transcript A achieved an MCD of 3.68, outperforming the LJSpeech benchmark (3.90), indicating high phonetic similarity to the target speaker.

Implicit Learning and Attention Bottlenecks

A key finding was that models trained on Transcript A learned to generate sighs and laughter despite their absence in the input text, suggesting they leveraged semantic cues in the surrounding speech. However, as transcripts became "cleaner" (Levels B and C), the discrepancy between the text and the "messy" audio caused the Transformer's attention mechanism to fail. This resulted in high CERs (exceeding 55%), highlighting a fundamental challenge in training standard TTS architectures on data where the audio contains significantly more information than the transcript.

Discussion and Future Work

DisfluencySpeech represents "acted" conversational speech, which ensures high audio quality but may lack some nuances of true spontaneous interaction. Additionally, the loss of interactive context from the original two-person dialogues is a limitation. Future work should explore more robust alignment techniques, such as hard monotonic alignments, to mitigate attention failures. Additionally, treating disfluencies as an auxiliary task rather than primary input could help models learn these nuances without requiring explicit textual representation.

Summary updated

BibTeX
@inproceedings{wang2024disfluencyspeech,
  author = {Kyra Wang and Dorien Herremans},
  title = {{DisfluencySpeech - Single-Speaker Conversational Speech Dataset with Paralanguage}},
  booktitle = {TENCON},
  year = {2024},
  url = {https://openreview.net/forum?id=wJeiHzH1sg},
}
Figure