DisfluencySpeech - Single-Speaker Conversational Speech Dataset with Paralanguage

Kyra Wang, Dorien Herremans

TL;DR

Researchers have introduced DisfluencySpeech, a new 10-hour, single-speaker dataset designed to help Text-to-Speech (TTS) models master "paralanguage"—the laughs, sighs, and "ums" that make human conversation feel real. By providing transcripts at varying levels of "cleanliness," the dataset enables models to learn how to predictively insert these nuances even when they aren't explicitly written in the text.

What Problem Does This Paper Address?

Most modern TTS systems sound remarkably human in isolation but fail to capture the messy reality of actual conversation. Real speech is full of paralanguage (non-lexical sounds like sighs or laughter) and disfluencies (filled pauses like "uh," false starts, and repetitions). These aren't just "noise"; they provide critical context for irony, hesitation, and turn-taking.

The challenge is that existing datasets are either "too clean" (like LJSpeech, which is based on audiobooks) or "too fragmented" (like Switchboard, which has many speakers but very little data per person). This makes it nearly impossible to train a model that can both speak clearly and sound naturally conversational.

Methods

The authors created DisfluencySpeech by having a single professional speaker (a 28-year-old Singaporean female) re-record nearly 10 hours of expressive utterances originally found in the Switchboard-1 Telephone Speech Corpus.

To test how well models can "guess" where paralanguage should go, they provided three types of transcripts:

  • Transcript A: Includes all spoken words, filled pauses ("uh"), and restarts. Non-speech events (laughter/sighs) are in the audio but removed from the text.
  • Transcript B: Removes filled pauses and discourse markers (e.g., "you know").
  • Transcript C: The "cleanest" version, removing false starts as well.
This figure illustrates how the same audio clip is represented across the three transcript levels.

The researchers then trained benchmark Transformer-based TTS models on these different levels to see if the models could implicitly learn to generate the missing paralinguistic features.

Datasets

  • DisfluencySpeech (New): 5,000 studio-quality clips (~10 hours) from a single speaker. It includes detailed annotations for five types of non-sentence elements.
  • Switchboard-1: The source material for the conversational scripts.
  • LJSpeech: Used as a baseline for "clean" speech performance.
Detailed statistics of the new DisfluencySpeech dataset.

Key Insights and Ideas

  • Implicit Learning: One of the most striking findings was that the model trained on Transcript A learned to generate sighs and laughter even though they weren't in the text. This suggests the model picked up on semantic cues in the sentences that typically precede a laugh or a sigh.
  • The Attention Bottleneck: While the models could handle some disfluency, they struggled with "missing" information. When the transcripts were too clean (Levels B and C) compared to the "messy" audio, the Transformer's attention mechanism broke down, leading to unintelligible speech.
  • Acting vs. Spontaneity: Because the speaker was reading a script that included disfluencies, the dataset represents "acted" conversational speech. While this ensures high audio quality, it differs slightly from the raw, unpredictable nature of spontaneous human interaction.

Key Results

The researchers evaluated their models using Mean Cepstral Distortion (MCD) and Character Error Rate (CER).

Performance metrics for the benchmark models compared to original datasets.
  • Audio Quality: The original DisfluencySpeech audio achieved a CER of 4.04%, very close to the gold-standard LJSpeech (3.30%), confirming its studio-quality status.
  • Model Performance: The model trained on Transcript A achieved an MCD of 3.68—actually better than the LJSpeech benchmark (3.90)—indicating high phonetic similarity to the original speaker.
  • Convergence Issues: Models trained on Transcripts B and C failed to converge effectively, resulting in very high CERs (over 55%). This highlights the difficulty of training standard TTS architectures on audio that contains significantly more information than the input text.

Limitations and Future Work

The authors acknowledge that by joining sub-utterances into single clips, some of the interactive context of the original two-person Switchboard dialogues is lost. Furthermore, the "acted" nature of the speech might lack some nuances of true spontaneity.

Future research could focus on using this dataset with more robust alignment techniques (like hard monotonic alignments) to prevent the attention failures seen in the cleaner transcript levels. Additionally, using these transcripts as an auxiliary task rather than the primary input might help models learn to model disfluencies without being explicitly "fed" them as text.

Summary updated

Abstract

Laughing, sighing, stuttering, and other forms of paralanguage do not contribute any direct lexical meaning to speech, but they provide crucial propositional context that aids semantic and pragmatic processes such as irony. It is thus important for artificial social agents to both understand and be able to generate speech with semantically-important paralanguage. Most speech datasets do not include transcribed non-lexical speech sounds and disfluencies, while those that do are typically multispeaker datasets where each speaker provides relatively little audio. This makes it challenging to train conversational Text-to-Speech (TTS) synthesis models that include such paralinguistic components. We thus present DisfluencySpeech, a studio-quality labeled English speech dataset with paralanguage. A single speaker recreates nearly 10 hours of expressive utterances from the Switchboard-1 Telephone Speech Corpus (Switchboard), simulating realistic informal conversations. To aid the development of a TTS model that is able to predictively synthesise paralanguage from text without such components, we provide three different transcripts at different levels of information removal (removal of non-speech events, removal of non-sentence elements, and removal of false starts), as well as benchmark TTS models trained on each of these levels.
BibTeX
@inproceedings{wang2024disfluencyspeech,
  author = {Kyra Wang and Dorien Herremans},
  title = {{DisfluencySpeech - Single-Speaker Conversational Speech Dataset with Paralanguage}},
  booktitle = {TENCON},
  year = {2024},
  url = {https://openreview.net/forum?id=wJeiHzH1sg},
}

Loading related content...

Figure