TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving

Wenqian Cui, Lei Zhu, Xiaohui Li, Zhihan Guo, Haoli Bai, Lu Hou, Irwin King

TL;DR

End-to-end full-duplex speech language models (FD-SLMs) enable natural voice conversations but often suffer from poor semantic coherence compared to text-only models. TurnGuide solves this by dynamically segmenting speech into conversational turns and interleaving text and speech chunks at the turn level. This approach injects the semantic intelligence of large language models into voice streams without disrupting natural acoustic flow or turn-taking dynamics.

What Problem Does This Paper Address?

Traditional voice assistants operate in a "walkie-talkie" mode: you speak, wait for the model to process, and then it responds. Full-Duplex Speech Language Models (FD-SLMs) aim for a more human-like experience by processing user and assistant audio streams simultaneously, allowing for real-time interactions like interruptions, backchanneling (e.g., saying "yeah" while the other person speaks), and overlapping speech.

Figure 1: Illustration of the two types of FD-SLMs. Cascaded FD-SLMs utilize a state predictor to guide predefined behaviors, while e2e FD-SLMs capture the complex dynamics of real-world double-channel dialogue data.

While end-to-end (e2e) FD-SLMs learn these complex conversational dynamics directly from real-world audio, their actual response quality often degrades. This happens because speech sequences are incredibly long, and high-quality spoken dialogue data is scarce compared to text.

Attempts to guide speech generation using text tokens usually introduce a new problem: if you insert text tokens exactly when the corresponding sound is produced (token-level interleaving), you fragment the text's meaning and risk disrupting the precise timing alignment required for fluid, real-time responses.

Methods

The authors introduce TurnGuide, a framework that manages text-speech integration through two main components:

  1. Dynamic Turn Segmentation and Alignment Framework: Instead of looking at speech as a continuous stream of individual words, TurnGuide uses Voice Activity Detection (VAD) to identify speech segments and merges them into Inter-Pausal Units (IPUs)—consecutive speech segments separated by pauses shorter than 0.5 seconds. It then uses an Automatic Speech Recognition (ASR) model (Whisper) to generate transcripts and extract word-level timestamps, grouping the text into distinct conversational "turns."

  2. Text-Guided Full Duplex Dialogue Modeling Framework: Built on top of the GLM-4-Voice backbone, TurnGuide applies two interleaving strategies:

  3. Channel-wise Interleaving: User and assistant audio channels are split into chunks (5 tokens each, ~400ms of audio) and interleaved alternately to capture cross-channel interactions.

  4. Turn-level Text-speech Interleaving: For each assistant turn, text chunks are generated just before their corresponding speech chunks (using a 5:5 token ratio). An "End of Chunk" (<EOC>) token is appended to help the model learn dynamic turn boundaries.
Figure 2: Illustration of the TurnGuide approach. The first part illustrates the multi-modal turn segmentation and alignment framework, which dynamically segments assistant speech into turns and aligns the text with speech turns. The second part shows the text-guided full duplex dialogue modeling framework with two interleaving strategies.

Datasets

  • Fisher Dataset: A standard double-channel telephone conversation corpus containing 2,000 hours of audio. The authors pre-processed the recordings into 120-second clips and explicitly defined a train/validation/test split to prevent test-set leakage in future research.
  • Speech QA Benchmarks: Web Questions, Llama Questions, and Trivia QA were used to evaluate whether the model preserved its core question-answering capabilities.

Key Insights and Ideas

  • Turn-Level Aggregation Beats Token-Level Fragmentation: Previous state-of-the-art models like Moshi insert text tokens precisely as they are sounded. TurnGuide shows that grouping text into turns provides a "dense semantic blueprint" that dramatically improves the model's ability to generate coherent, meaningful responses.
  • Robustness to Timestamp Noise: Token-level alignment relies heavily on perfect word-level timestamps from ASR systems, which are notoriously noisy in practice. Because TurnGuide only requires the start time of a text chunk, it is significantly more robust to alignment errors.
  • Prioritizing Text Generation Boosts Speech Quality: By adjusting the training loss weights to favor text tokens over speech tokens (e.g., a 3:1 ratio), the model is forced to focus more on semantic planning, which directly translates to higher-quality spoken outputs.

Key Results

TurnGuide was evaluated using GPT-4o to score the semantic meaningfulness and coherence of the generated dialogues (scaled from 0 to 10), alongside perplexity metrics and human absolute ratings.

  • Superior Semantics: In unconditional generation, TurnGuide (with a 3:1 loss ratio) achieved a semantic score of 9.03, vastly outperforming the speech-only baseline (6.94) and the Moshi token-level training strategy (6.76).
  • Lower Perplexity: Across multiple evaluation language models, TurnGuide consistently achieved lower perplexity, confirming its superior semantic modeling capabilities.

Perplexity Evaluation Across Different Models Semantic Evaluation Results by GPT-4o

  • Preserved Turn-Taking & Low Latency: TurnGuide maintained state-of-the-art performance on Full-Duplex-Bench for handling user interruptions and backchanneling. Crucially, it achieved a theoretical end-to-end interaction latency of 0.69 seconds, well within the acceptable threshold for real-time human conversation.

Limitations and Future Work

  • Dataset Homogeneity: TurnGuide was primarily validated on the Fisher dataset. Because Fisher consists entirely of telephone conversations, it may not fully capture the stylistic diversity of other real-world conversational settings.
  • Vocoder Constraints: While a 5-token chunk size is optimal for low latency, the underlying GLM-4-Voice vocoder requires processing at least 10 tokens together to maintain acoustic stability. Optimizing the vocoder for smaller chunk sizes remains an open area for future work.

Summary updated

Abstract

Full-Duplex Speech Language Models (FD-SLMs) are specialized foundation models designed to enable natural, real-time spoken interactions by modeling complex conversational turn-taking such as interruptions, backchannels, and overlapping speech. End-to-end (e2e) FD-SLMs leverage real-world double-channel conversational data to capture nuanced two-speaker dialogue patterns for human-like interactions, but their conversational abilities often degrade compared to pure-text conversation due to prolonged speech sequences and limited high-quality spoken dialogue data. Although interleaved text-speech generation could mitigate this degradation, integrating discrete text tokens into continuous double-channel audio streams could disrupt the precise time alignment required for fluid interaction. To address this, we propose TurnGuide, a novel text-speech interleaved generation approach for e2e FD-SLMs that dynamically segments assistant speech into dialogue turns and interleaves turn-level text and speech generation. This approach allows FD-SLMs to integrate the semantic intelligence of LLMs without compromising the natural acoustic flow. Extensive experiments show that TurnGuide not only significantly improves e2e FD-SLMs to produce semantically meaningful, coherent speech but also achieves state-of-the-art performance on various turn-taking events. Demos are available at https://dreamtheater123.github.io/TurnGuide-Demo/. Code is available at https://github.com/dreamtheater123/TurnGuide.
BibTeX
@misc{cui2025turnguide,
  author = {Wenqian Cui and Lei Zhu and Xiaohui Li and Zhihan Guo and Haoli Bai and Lu Hou and Irwin King},
  title = {{TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving}},
  year = {2025},
  eprint = {2508.07375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.07375},
}
Figure