Modeling Spontaneous Spoken Dialogue and Disfluencies

This direction is split into other directions: spontaneous speech gap. Its last version stays readable and citable here.

Version 1 of 156 papers61 conceptsfrozen 2026-10-04

Most speech AI today sounds like a news anchor reading a teleprompter: perfectly paced, grammatically flawless, and strictly turn-based. But real human conversation is "messy." We hesitate with "um" and "uh," we interrupt each other, we use backchannels like "uh-huh" to show we are listening, and we frequently restart our sentences mid-thought.

Researchers are shifting away from "clean" speech models toward systems that embrace these spontaneous disfluencies. The goal is to move beyond simple command-and-response interfaces toward agents that can navigate the rhythmic and linguistic complexities of natural human interaction.

The Problem with Clean Data

The primary hurdle in conversational AI is a massive data mismatch. Most large-scale speech datasets are derived from audiobooks, news broadcasts, or scripted prompts. As Shengpeng Ji and colleagues note in WavChat: A Survey of Spoken Dialogue Models, relying on synthetic or read speech fails to capture the nuances of real-world conversation.

When a model is trained only on "clean" text, it views a hesitation like "um" as noise to be filtered out. However, in high-stakes environments like medical dictation, Serguei V. S. Pakhomov found that speech recognizers often try too hard to make sense of these sounds, hallucinating words like "thumb" or "arm" instead of recognizing a pause. This suggests that for a model to be robust, it must be "polluted" with the very disfluencies it will encounter in the wild.

The Anatomy of Spontaneous Speech

Spontaneous speech isn't just "broken" formal speech; it has its own structural logic. Researchers categorize these features into several key mechanisms:

  • Filled Pauses and Fillers: Sounds like "um" and "uh" act as placeholders, signaling that a speaker is thinking but intends to keep the floor. Xu Tan and the creators of AdaSpeech 3 demonstrate that including these fillers significantly increases the perceived naturalness of synthesized speech.
  • Disfluencies and Repairs: These include stutters, mid-sentence corrections, and restarts. Bill Byrne argues in the Taskmaster-1 paper that "self-dialogue" (where one person writes both sides of a chat) often misses these corrections, making the resulting models brittle when faced with a user who changes their mind mid-sentence.
  • Temporal Dynamics: Real conversation is "full-duplex," meaning both parties can speak at once. Bandhav Veluri and the team behind SyncLLM highlight that standard LLMs struggle with this because they wait for a "silence" token to respond. True spontaneity requires handling interruptions and backchanneling in real-time.

Bridging the Gap with Naturalistic Corpora

To solve the data scarcity, researchers are moving away from scripted media. While movie scripts provide rich world knowledge, Rafael E. Banchs and Kazuyuki Matsumoto point out that scripts still omit the messy overlaps and non-verbal cues of genuine talk.

The field is now prioritizing "found" spontaneous data:

  • CASPER: C. Xiao and colleagues introduced CASPER, a 200-hour dataset of genuine conversations between acquaintances, specifically to capture real-world disfluencies.
  • J-CHAT: For Japanese, Wataru Nakata developed J-CHAT by mining podcasts and YouTube to capture the back-and-forth rhythm of natural dialogue.
  • MagicData-RAMC: Yi-Fan Chen released MagicData-RAMC to provide a Mandarin benchmark that includes colloquialisms and disfluencies missing from standard "reading-style" datasets.

Toward Unified Spoken Dialogue Models

The traditional "cascaded" approach—where an ASR model converts speech to text, an LLM generates a text response, and a TTS model speaks it—is inherently limited. It loses the prosody (the "how" of speaking) at every step.

Delong Chen and others are advocating for Joint Modeling of Dialogue Response and Speech Synthesis. By training a single model to "think how to respond" and "think how to speak" simultaneously, the AI can learn to use fillers and pauses strategically. For instance, a model might insert an "um" when it is performing a complex knowledge retrieval, mimicking the way humans use pauses to manage cognitive load.

This shift is not just about making AI sound more human; it is about making it more functional. As Haonan Chen demonstrates with CONVAUG, models that understand the unpredictable, fragmented nature of human speech are significantly better at retrieving information and sustaining helpful, multi-turn interactions in the real world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_modeling_spontaneous_spoken_dialogue_and_3bf99182,
  title = {Modeling Spontaneous Spoken Dialogue and Disfluencies},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-spontaneous-spoken-dialogue-and-disfluencies/rdr_3bf991825def44f79c270f99af5311e4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_modeling_spontaneous_spoken_dialogue_and_3bf99182_v1,
  title = {Modeling Spontaneous Spoken Dialogue and Disfluencies},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-spontaneous-spoken-dialogue-and-disfluencies/rdr_3bf991825def44f79c270f99af5311e4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}