Traditional text-to-speech (TTS) systems excel at reading structured, formal text—such as audiobooks or news broadcasts—with pristine clarity. However, when these systems are deployed in interactive conversational agents or podcasts, they often fall into an "uncanny valley." They sound artificial not because their audio quality is poor, but because their delivery is too perfect.
Real human conversation is inherently messy. It is filled with hesitations, word prolongations, repetitions, breathing pauses, and non-lexical fillers like "uh" and "um." As noted by Le et al. and Mitsui et al., models trained exclusively on clean, read-style speech lack these acoustic irregularities, making synthesized multi-speaker interactions sound sterile and detached from natural human behavior.
The Communicative Value of Speech Messiness
To build systems that sound truly spontaneous, we must first understand why humans use disfluencies. In cognitive linguistics and acoustic prosody, these imperfections are not random errors; they are functional signals that carry critical communicative value.
A filled pause or a sudden elongation of a vowel signals cognitive load, indicating that the speaker is planning their next phrase or processing complex information. For instance, Dall et al. demonstrated that human listeners have highly consistent preferences for where filled pauses are placed, confirming that their distribution follows structured linguistic patterns rather than random insertion.
Furthermore, disfluencies modulate how a listener interprets the speaker's state of mind. Székely et al. and Kirkland et al. showed that the interplay between vocal effort, speech rate, and hesitation disfluencies directly dictates how listeners perceive a speaker's confidence and uncertainty. When a TTS system strips away these elements to achieve perfect fluency, it inadvertently strips away the social and emotional nuances necessary for authentic dialogue.
From Heuristics to Explicit Disfluency Modeling
Early attempts to bridge this naturalness gap relied on rule-based injection. Marge et al. demonstrated that programmatically inserting filled pauses and word repetitions into text-based dialogue logs significantly boosted the perceived naturalness of synthetic voices. However, static heuristic rules do not scale and fail to capture the intricate relationship between linguistic context and acoustic prosody.
To make these behaviors dynamic, modern architectures explicitly integrate disfluency modeling into the acoustic generation pipeline. The mechanism relies on conditioning the acoustic model on structural and conversational markers:
- Contextual Tagging: Yamashita et al. showed that feeding deep neural networks rich, manually annotated contextual data—such as explicit disfluency tags and clause boundaries—allows acoustic models to accurately predict the duration and pitch variations of non-lexical speech elements. This prevents the model from smoothing over hesitations.
- Predictive Placement: Instead of relying on human annotation, systems like the one proposed by Hu et al. utilize a BERT-based disfluency predictor to identify optimal linguistic locations for filled pauses and prolongations based on the text context. By applying a frequency threshold to these predictions, the system allows for adjustable levels of speech fluency during synthesis.
By treating disfluencies as learnable stylistic elements conditioned on conversational context, as explored by Lei et al., acoustic models learn the precise rhythmic breaks and temporal dynamics that characterize human hesitation.
Emergent Spontaneity via Language Models
The latest paradigm shifts away from explicit tagging toward leveraging large language models (LLMs) to let conversational spontaneity emerge naturally. Because LLMs can capture the underlying cognitive patterns of human thought, they can simulate realistic hesitations based on linguistic complexity.
Zhang et al. demonstrated that LLMs can place hesitations and repetitions in linguistically appropriate contexts (such as before complex clauses or specific parts of speech), mimicking natural cognitive load far better than rigid, rule-based synthetic data generation.
On the acoustic side, scaling up autoregressive and diffusion-based language models allows for the direct generation of these complex behaviors from data. Li et al. combined a large-scale language model backbone with a framework capable of modeling 19 distinct spontaneous behaviors and fine-grained prosody, allowing explicit control over how these behaviors are expressed.
To bypass the need for clean, meticulously annotated datasets, Ju et al. introduced an "ASR-proxy" approach. By training speech synthesis models on scripts that mimic Automatic Speech Recognition (ASR) transcripts, conversational spontaneity—including filler words, hesitations, and breathing patterns—emerges organically. Crucially, the authors found that the inclusion of these messy conversational elements does not compromise the intelligibility of the generated speech, but provides significant improvements in subjective measures of spontaneity and coherence.
Key Foundational Work
Computer Simulation of Spontaneous Speech Production (1984) — contributed 3 concept elements to this direction.