Traditional voice assistants operate like a rigid game of walkie-talkie tag: a user speaks, the system waits for a fixed period of silence to detect that the turn has ended, and only then does it begin processing a response. This sequential, "half-duplex" interaction pattern creates an artificial, robotic cadence.
In contrast, natural human conversation is fluid, continuous, and overlapping. Humans listen and think while simultaneously speaking, allowing them to throw in quick verbal affirmations ("mhm", "yeah"), handle sudden mid-sentence topic shifts, or immediately halt when interrupted.
To bridge this gap, researchers are shifting toward full-duplex spoken dialogue modeling. This paradigm treats listening and speaking not as separate, alternating phases, but as concurrent processes. By moving away from rigid turn-taking boundaries, modern speech systems can achieve real-time responsiveness and human-like conversational fluidity.
The Half-Duplex Bottleneck
To understand why full-duplex modeling is necessary, we must look at the structural limitations of conventional spoken dialogue systems. Most standard voice architectures rely on a "cascaded" pipeline: a Voice Activity Detection (VAD) module flags when the user stops talking, an Automatic Speech Recognition (ASR) model transcribes the audio, a Large Language Model (LLM) generates a text response, and a Text-to-Speech (TTS) engine synthesizes the final audio.
As noted by Wenqian Cui et al., this cascaded approach suffers from high cumulative latency and completely strips out non-verbal dynamics. Because the system is structurally blind to incoming audio while it is generating a response, it cannot handle "barge-ins"—scenarios where the user interrupts the AI mid-sentence. As described by Maosong Sun et al., this forces a strict separation between user input and system response, preventing the model from adapting dynamically to real-time feedback.
From Incremental Processing to Streaming Tokens
The foundational idea behind overcoming this bottleneck is incremental processing. Instead of waiting for an entire utterance to finish before initiating semantic analysis, early frameworks sought to process language in smaller, real-time segments.
Gabriel Skantze et al. and Timo Baumann et al. demonstrated that breaking down speech recognition and synthesis into incremental chunks allows a system to react to partial inputs. This enables conversational behaviors like mid-utterance feedback and rapid turn coordination.
Modern Speech Language Models (SpeechLMs) scale this concept by bypassing the text-based cascade entirely, encoding raw audio directly into continuous streams of discrete tokens. To achieve full-duplex communication within a single model architecture, researchers use two primary strategies to handle concurrent listening and speaking channels:
- Interleaved and Flattened Sequences: Qinglin Zhang demonstrates that a standard text-based, decoder-only LLM can be adapted for full-duplex voice interaction by "flattening" interleaved streams of input (user) and output (system) tokens into a single sequence.
- Time-Slicing and Dual-Channel Training: The framework proposed by Maosong Sun et al. breaks conversations down into tiny time slices. The model processes the incoming audio stream and generates the outgoing response stream simultaneously, maintaining a dual-state context.
Because the model's self-attention mechanism operates across both channels at every step, it can compute cross-stream dependencies on the fly. The model is literally "listening" to the user's incoming tokens while calculating its next output token.
Half-Duplex (Sequential):
User: [--- Utterance ---]
VAD: [Silence]
System: [--- Processing & Response ---]
Full-Duplex (Concurrent/Streaming):
User: [--- Token 1 ---][--- Token 2 ---][--- Token 3 (Interruption) ---]
System: [--- Response Token 1 ---][--- Halts / Adapts ---]
Managing State and Interruption Autoregressively
If a model is constantly processing parallel streams, how does it decide when to speak, when to stay silent, and when to stop talking?
Instead of relying on an external, rule-based VAD controller to cut off the audio, full-duplex systems internalize conversation state management as a next-token prediction task. The authors of Cocktail-Talker introduce explicit turn-action tokens—such as <|respond|>, <|listen|>, and <|ignore|>—directly into the model's vocabulary. This allows the system to navigate noisy, multi-speaker environments by predicting whether it should actively participate or remain a passive listener.
Similarly, Peng Wang et al. fine-tuned an 8-billion parameter LLM to manage turn transitions dynamically. When a system is speaking and the user begins to talk over it, the streaming audio encoder injects the user's new audio tokens into the model's context window.
Because the model has been trained on interleaved dialogue histories where user interruptions occur, the sudden appearance of user speech tokens shifts the autoregressive probability distribution. The model naturally predicts an interruption or floor-yielding token, which instantly triggers the system to halt its text-to-speech synthesis pipeline.
Alignment for Natural Conversational Timing
Building a model capable of full-duplex communication is only half the battle; it must also be aligned to use these capabilities appropriately. Without careful tuning, a full-duplex model might suffer from "greedy" interruptions, cutting users off prematurely during natural pauses, or failing to provide backchannels when the user expects reassurance.
To inject realistic social cues, Sehun Lee et al. introduced datasets annotated with human-like conversational behaviors, training models to naturally emit filler words and backchannel feedback ("mhm", "yeah") without fully taking over the conversational floor.
Furthermore, traditional reinforcement learning techniques fail here because standard evaluation metrics treat dialogue as discrete, static turns. As highlighted by Yizhou Peng et al., evaluating full-duplex systems requires specialized pipelines that simulate asynchronous interruptions and measure precise reaction delays.
To address this, Anne Wu et al. developed alignment frameworks using length-normalized Direct Preference Optimization (DPO-LN) on preference pairs generated from real user interactions. This optimization explicitly penalizes awkward response latencies and inappropriate interruptions, teaching the model not just what to say, but exactly when it is socially appropriate to say it.