Synthesizing Spontaneous Speech Through Conversational Disfluencies

97 concepts53 papers2 proposals

Traditional text-to-speech (TTS) systems excel at reading structured, formal text—such as audiobooks or news broadcasts—with pristine clarity. However, when these systems are deployed in interactive conversational agents or podcasts, they often fall into an "uncanny valley." They sound artificial not because their audio quality is poor, but because their delivery is too perfect.

Real human conversation is inherently messy. It is filled with hesitations, word prolongations, repetitions, breathing pauses, and non-lexical fillers like "uh" and "um." As noted by Le et al. and Mitsui et al., models trained exclusively on clean, read-style speech lack these acoustic irregularities, making synthesized multi-speaker interactions sound sterile and detached from natural human behavior.

The Communicative Value of Speech Messiness

To build systems that sound truly spontaneous, we must first understand why humans use disfluencies. In cognitive linguistics and acoustic prosody, these imperfections are not random errors; they are functional signals that carry critical communicative value.

A filled pause or a sudden elongation of a vowel signals cognitive load, indicating that the speaker is planning their next phrase or processing complex information. For instance, Dall et al. demonstrated that human listeners have highly consistent preferences for where filled pauses are placed, confirming that their distribution follows structured linguistic patterns rather than random insertion.

Furthermore, disfluencies modulate how a listener interprets the speaker's state of mind. Székely et al. and Kirkland et al. showed that the interplay between vocal effort, speech rate, and hesitation disfluencies directly dictates how listeners perceive a speaker's confidence and uncertainty. When a TTS system strips away these elements to achieve perfect fluency, it inadvertently strips away the social and emotional nuances necessary for authentic dialogue.

From Heuristics to Explicit Disfluency Modeling

Early attempts to bridge this naturalness gap relied on rule-based injection. Marge et al. demonstrated that programmatically inserting filled pauses and word repetitions into text-based dialogue logs significantly boosted the perceived naturalness of synthetic voices. However, static heuristic rules do not scale and fail to capture the intricate relationship between linguistic context and acoustic prosody.

To make these behaviors dynamic, modern architectures explicitly integrate disfluency modeling into the acoustic generation pipeline. The mechanism relies on conditioning the acoustic model on structural and conversational markers:

  1. Contextual Tagging: Yamashita et al. showed that feeding deep neural networks rich, manually annotated contextual data—such as explicit disfluency tags and clause boundaries—allows acoustic models to accurately predict the duration and pitch variations of non-lexical speech elements. This prevents the model from smoothing over hesitations.
  2. Predictive Placement: Instead of relying on human annotation, systems like the one proposed by Hu et al. utilize a BERT-based disfluency predictor to identify optimal linguistic locations for filled pauses and prolongations based on the text context. By applying a frequency threshold to these predictions, the system allows for adjustable levels of speech fluency during synthesis.

By treating disfluencies as learnable stylistic elements conditioned on conversational context, as explored by Lei et al., acoustic models learn the precise rhythmic breaks and temporal dynamics that characterize human hesitation.

Emergent Spontaneity via Language Models

The latest paradigm shifts away from explicit tagging toward leveraging large language models (LLMs) to let conversational spontaneity emerge naturally. Because LLMs can capture the underlying cognitive patterns of human thought, they can simulate realistic hesitations based on linguistic complexity.

Zhang et al. demonstrated that LLMs can place hesitations and repetitions in linguistically appropriate contexts (such as before complex clauses or specific parts of speech), mimicking natural cognitive load far better than rigid, rule-based synthetic data generation.

On the acoustic side, scaling up autoregressive and diffusion-based language models allows for the direct generation of these complex behaviors from data. Li et al. combined a large-scale language model backbone with a framework capable of modeling 19 distinct spontaneous behaviors and fine-grained prosody, allowing explicit control over how these behaviors are expressed.

To bypass the need for clean, meticulously annotated datasets, Ju et al. introduced an "ASR-proxy" approach. By training speech synthesis models on scripts that mimic Automatic Speech Recognition (ASR) transcripts, conversational spontaneity—including filler words, hesitations, and breathing patterns—emerges organically. Crucially, the authors found that the inclusion of these messy conversational elements does not compromise the intelligibility of the generated speech, but provides significant improvements in subjective measures of spontaneity and coherence.

Key Foundational Work

Computer Simulation of Spontaneous Speech Production (1984) — contributed 3 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 56 papers
  2. Version 1 · 2026-10-04 · 55 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_spontaneous_speech_synthesis_7abb6c5b,
  title = {spontaneous speech synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spontaneous-speech-synthesis/rdr_7abb6c5b98074c6f8125bfb9f223c630},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_spontaneous_speech_synthesis_7abb6c5b_v2,
  title = {spontaneous speech synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spontaneous-speech-synthesis/rdr_7abb6c5b98074c6f8125bfb9f223c630/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake