Incremental Processing for Low Latency Spoken Dialogue

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2171 papers243 conceptsfrozen 2026-10-04

In a typical conversation with a voice assistant, there is a palpable "awkward pause." You finish speaking, the system waits for a second of silence to ensure you are done, and only then does it begin the heavy lifting of speech recognition, intent parsing, and response generation. This rigid, turn-based rhythm is a byproduct of the linear pipeline architecture that has dominated Natural Language Understanding (NLU) for decades.

In contrast, human listeners do not wait for silence. As Michael K. Tanenhaus and colleagues demonstrated in their seminal work on Eye Movements and Spoken Language Comprehension, the human brain integrates phonetic, visual, and pragmatic cues the instant they are heard. We begin to look at a target object within 200 milliseconds of hearing its first syllable—long before the word itself is finished.

To bridge this gap, researchers are shifting toward incremental processing: a paradigm where systems "think" while they listen, overlapping computation with user speech to achieve the fluid, "full-duplex" feel of human interaction.

Breaking the Linear Pipeline

The traditional approach treats NLU as a sequence: first syntax, then semantics, then pragmatics. However, Robert J. Bobrow and Bonnie Webber argued as early as 1981 that this is fundamentally inefficient. In Some Issues in Parsing and Natural Language Understanding, they proposed the Principle of Continually Available Output (CAO). Instead of waiting for a full sentence, modular processors should provide partial outputs in real-time.

By processing word-by-word, a system can resolve ambiguity early. For example, if a user says "Pick up the red...", the system can narrow down the potential objects before the noun is even spoken. This is the core of Incremental Significant-Utterance-Sequence Search (ISSS), which updates the system's internal belief state continuously, drastically reducing the time it takes to complete a task.

Anticipation and Speculative Pre-filling

If the goal is to eliminate latency entirely, the system must do more than just process quickly—it must predict the future. Recent work on SyncLLM by Bandhav Veluri and others uses "synchronization tokens" to track real-world time, allowing the model to predict the user's immediate future speech. By appending these estimations to the current context, the model can prepare its response before the audio data has fully arrived.

This "speculative pre-filling" is also a cornerstone of the LongCat-Flash-Omni architecture. By using voice activity detection to predict when a user is about to finish, the system triggers the pre-filling of its response buffer while the user is still speaking their final words. This effectively masks the computational overhead of the Large Language Model (LLM).

Turn-Taking and Pragmatic Completeness

A major challenge in incremental systems is knowing when to jump in without being "greedy" or rude. Traditional systems rely on silence, but silence is ambiguous—it could be a pause for thought or the end of a turn.

Erik Ekstedt and Gabriel Skantze addressed this with TurnGPT, which uses linguistic cues to predict "pragmatic completeness." Their follow-up work, RC-TurnGPT, adds a layer of "contextual patience" by conditioning turn-taking on the system's intended response. If the system realizes its prepared answer better addresses a later part of the user's utterance, it can suppress the urge to interrupt prematurely.

Thinking While Listening

The most recent evolution of this idea involves moving complex reasoning into the listening phase. In Can Speech LLMs Think while Listening?, Desh Raj and colleagues introduced a paradigm where the model performs text-based Chain-of-Thought (CoT) reasoning concurrently with the audio stream.

By initiating reasoning as soon as the input reaches "semantic completeness"—rather than waiting for physical silence—they achieved a 70% reduction in latency. This suggests that the future of conversational AI lies in full-duplex systems, like those proposed by Peng Wang and the Meituan LongCat Team, where the boundary between "listening" and "responding" is entirely dissolved.

Papers in this version

Showing 60 of 171 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 211 papers
  2. Version 1 · 2026-10-04 · 171 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_incremental_speech_processing_bc8ac91f,
  title = {incremental speech processing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incremental-speech-processing/rdr_bc8ac91f606e467a889a1efdc4487481},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_incremental_processing_for_low_latency_s_bc8ac91f_v1,
  title = {Incremental Processing for Low Latency Spoken Dialogue},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incremental-processing-for-low-latency-spoken-dialogue/rdr_bc8ac91f606e467a889a1efdc4487481/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}