Modeling Spontaneous Behaviors in Conversational Speech Synthesis

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 255 papers76 conceptsfrozen 2026-10-04

Traditional Text-to-Speech (TTS) systems are remarkably good at reading books. They produce clear, rhythmic, and grammatically perfect audio. However, when these same models are used in a conversational setting—like a voice assistant or a podcast—they often sound "uncanny" or robotic.

The reason is that human conversation is fundamentally messy. We don't just speak words; we hesitate, prolong sounds, breathe audibly, and use fillers like "um" and "uh." Research in spontaneous speech synthesis aims to bridge this gap by explicitly modeling the disfluencies and non-lexical cues that characterize natural human interaction.

The Reading Style vs Spontaneous Style Gap

Most neural TTS models, as noted in A Survey on Neural Speech Synthesis by Xu Tan and colleagues, are trained on high-quality audiobook datasets. While these datasets provide a clean signal for learning phonetics, they lack the "spontaneous" style of everyday talk.

In a formal reading style, the tempo is stable and the delivery is fluent. In contrast, spontaneous speech is defined by disfluencies:

  • Filled Pauses: Non-lexical sounds like "um," "uh," or "ah" used to hold a turn in conversation.
  • Prolongations: Stretching out a word (e.g., "theee...") while thinking of the next one.
  • Non-lexical Cues: Breaths, laughter, and lip smacks that provide paralinguistic context.

As Na Hu and Lei Xie argue in Controllable Context-Aware Conversational Speech Synthesis, omitting these behaviors makes synthetic speech sound overly formal and sterile.

Predicting the Messiness

To synthesize these behaviors, researchers have moved beyond simple text-to-audio mapping. The challenge is that disfluencies are not usually written in the input text. Models must therefore predict where a human would naturally hesitate.

AdaSpeech 3 addresses this by using a Mixture-of-Experts (MoE) duration predictor. Because spontaneous speech has irregular pacing, a single duration model often averages out the variations, leading to robotic rhythms. The MoE approach allows the model to choose between different rhythmic "experts," better capturing the dynamic tempo changes of a real person.

Other approaches, such as those by Weiqin Li in Spontaneous Style Text-to-Speech Synthesis, categorize dozens of distinct spontaneous behaviors. By treating these as learnable stylistic elements conditioned on linguistic context, systems can give users explicit control over how "disfluent" the output should be.

Learning from In-the-Wild Data

A major bottleneck in this field is data. Clean, annotated conversational data is rare. Researchers are now finding ways to leverage "in-the-wild" audio, such as podcasts or call recordings.

Voicebox, developed by Matthew Le and others, uses a flow-matching objective to learn from 50,000 hours of diverse audio. Instead of predicting the next token, it learns to "infill" speech, which naturally allows it to handle the non-linearities of conversation.

Similarly, MoonCast introduces an ASR-proxy approach. By training on scripts that look like raw Automatic Speech Recognition transcripts—complete with the errors and fillers typical of human speech—the model learns to generate those same spontaneous patterns. As Zeqian Ju demonstrates, this results in the natural emergence of breathing patterns and hesitations that are typically lost in models trained on "clean" text.

Scaling to Multi-talker Interactions

The ultimate goal of this research is to move from isolated monologues to realistic dialogues. This requires modeling acoustic entrainment—the way speakers subconsciously align their pitch and tempo with one another.

Models like CoVoMix by Lei He and Shujie Liu are designed to handle multi-speaker dynamics, including natural overlaps and interjections. By predicting simultaneous semantic streams, these frameworks capture the rhythmic "give and take" of a real conversation.

As we move toward long-form generation, systems like SpeechSSM and VibeVoice are pushing the boundaries of stability, enabling minutes of coherent, expressive audio that sounds less like a machine reading a script and more like a human sharing a story.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 56 papers
  2. Version 1 · 2026-10-04 · 55 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_spontaneous_speech_synthesis_7abb6c5b,
  title = {spontaneous speech synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spontaneous-speech-synthesis/rdr_7abb6c5b98074c6f8125bfb9f223c630},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_modeling_spontaneous_behaviors_in_conver_7abb6c5b_v1,
  title = {Modeling Spontaneous Behaviors in Conversational Speech Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-spontaneous-behaviors-in-conversational-speech-synthesis/rdr_7abb6c5b98074c6f8125bfb9f223c630/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}