TL;DR
Researchers have introduced DisfluencySpeech, a new 10-hour, single-speaker dataset designed to help Text-to-Speech (TTS) models master "paralanguage"—the laughs, sighs, and "ums" that make human conversation feel real. By providing transcripts at varying levels of "cleanliness," the dataset enables models to learn how to predictively insert these nuances even when they aren't explicitly written in the text.
What Problem Does This Paper Address?
Most modern TTS systems sound remarkably human in isolation but fail to capture the messy reality of actual conversation. Real speech is full of paralanguage (non-lexical sounds like sighs or laughter) and disfluencies (filled pauses like "uh," false starts, and repetitions). These aren't just "noise"; they provide critical context for irony, hesitation, and turn-taking.
The challenge is that existing datasets are either "too clean" (like LJSpeech, which is based on audiobooks) or "too fragmented" (like Switchboard, which has many speakers but very little data per person). This makes it nearly impossible to train a model that can both speak clearly and sound naturally conversational.
Methods
The authors created DisfluencySpeech by having a single professional speaker (a 28-year-old Singaporean female) re-record nearly 10 hours of expressive utterances originally found in the Switchboard-1 Telephone Speech Corpus.
To test how well models can "guess" where paralanguage should go, they provided three types of transcripts:
- Transcript A: Includes all spoken words, filled pauses ("uh"), and restarts. Non-speech events (laughter/sighs) are in the audio but removed from the text.
- Transcript B: Removes filled pauses and discourse markers (e.g., "you know").
- Transcript C: The "cleanest" version, removing false starts as well.
The researchers then trained benchmark Transformer-based TTS models on these different levels to see if the models could implicitly learn to generate the missing paralinguistic features.
Datasets
- DisfluencySpeech (New): 5,000 studio-quality clips (~10 hours) from a single speaker. It includes detailed annotations for five types of non-sentence elements.
- Switchboard-1: The source material for the conversational scripts.
- LJSpeech: Used as a baseline for "clean" speech performance.
Key Insights and Ideas
- Implicit Learning: One of the most striking findings was that the model trained on Transcript A learned to generate sighs and laughter even though they weren't in the text. This suggests the model picked up on semantic cues in the sentences that typically precede a laugh or a sigh.
- The Attention Bottleneck: While the models could handle some disfluency, they struggled with "missing" information. When the transcripts were too clean (Levels B and C) compared to the "messy" audio, the Transformer's attention mechanism broke down, leading to unintelligible speech.
- Acting vs. Spontaneity: Because the speaker was reading a script that included disfluencies, the dataset represents "acted" conversational speech. While this ensures high audio quality, it differs slightly from the raw, unpredictable nature of spontaneous human interaction.
Key Results
The researchers evaluated their models using Mean Cepstral Distortion (MCD) and Character Error Rate (CER).
- Audio Quality: The original DisfluencySpeech audio achieved a CER of 4.04%, very close to the gold-standard LJSpeech (3.30%), confirming its studio-quality status.
- Model Performance: The model trained on Transcript A achieved an MCD of 3.68—actually better than the LJSpeech benchmark (3.90)—indicating high phonetic similarity to the original speaker.
- Convergence Issues: Models trained on Transcripts B and C failed to converge effectively, resulting in very high CERs (over 55%). This highlights the difficulty of training standard TTS architectures on audio that contains significantly more information than the input text.
Limitations and Future Work
The authors acknowledge that by joining sub-utterances into single clips, some of the interactive context of the original two-person Switchboard dialogues is lost. Furthermore, the "acted" nature of the speech might lack some nuances of true spontaneity.
Future research could focus on using this dataset with more robust alignment techniques (like hard monotonic alignments) to prevent the attention failures seen in the cleaner transcript levels. Additionally, using these transcripts as an auxiliary task rather than the primary input might help models learn to model disfluencies without being explicitly "fed" them as text.
Summary updated