TL;DR
End-to-end full-duplex speech language models (FD-SLMs) enable natural voice conversations but often suffer from poor semantic coherence compared to text-only models. TurnGuide solves this by dynamically segmenting speech into conversational turns and interleaving text and speech chunks at the turn level. This approach injects the semantic intelligence of large language models into voice streams without disrupting natural acoustic flow or turn-taking dynamics.
What Problem Does This Paper Address?
Traditional voice assistants operate in a "walkie-talkie" mode: you speak, wait for the model to process, and then it responds. Full-Duplex Speech Language Models (FD-SLMs) aim for a more human-like experience by processing user and assistant audio streams simultaneously, allowing for real-time interactions like interruptions, backchanneling (e.g., saying "yeah" while the other person speaks), and overlapping speech.
While end-to-end (e2e) FD-SLMs learn these complex conversational dynamics directly from real-world audio, their actual response quality often degrades. This happens because speech sequences are incredibly long, and high-quality spoken dialogue data is scarce compared to text.
Attempts to guide speech generation using text tokens usually introduce a new problem: if you insert text tokens exactly when the corresponding sound is produced (token-level interleaving), you fragment the text's meaning and risk disrupting the precise timing alignment required for fluid, real-time responses.
Methods
The authors introduce TurnGuide, a framework that manages text-speech integration through two main components:
-
Dynamic Turn Segmentation and Alignment Framework: Instead of looking at speech as a continuous stream of individual words, TurnGuide uses Voice Activity Detection (VAD) to identify speech segments and merges them into Inter-Pausal Units (IPUs)—consecutive speech segments separated by pauses shorter than 0.5 seconds. It then uses an Automatic Speech Recognition (ASR) model (Whisper) to generate transcripts and extract word-level timestamps, grouping the text into distinct conversational "turns."
-
Text-Guided Full Duplex Dialogue Modeling Framework: Built on top of the GLM-4-Voice backbone, TurnGuide applies two interleaving strategies:
-
Channel-wise Interleaving: User and assistant audio channels are split into chunks (5 tokens each, ~400ms of audio) and interleaved alternately to capture cross-channel interactions.
- Turn-level Text-speech Interleaving: For each assistant turn, text chunks are generated just before their corresponding speech chunks (using a 5:5 token ratio). An "End of Chunk" (
<EOC>) token is appended to help the model learn dynamic turn boundaries.
Datasets
- Fisher Dataset: A standard double-channel telephone conversation corpus containing 2,000 hours of audio. The authors pre-processed the recordings into 120-second clips and explicitly defined a train/validation/test split to prevent test-set leakage in future research.
- Speech QA Benchmarks: Web Questions, Llama Questions, and Trivia QA were used to evaluate whether the model preserved its core question-answering capabilities.
Key Insights and Ideas
- Turn-Level Aggregation Beats Token-Level Fragmentation: Previous state-of-the-art models like Moshi insert text tokens precisely as they are sounded. TurnGuide shows that grouping text into turns provides a "dense semantic blueprint" that dramatically improves the model's ability to generate coherent, meaningful responses.
- Robustness to Timestamp Noise: Token-level alignment relies heavily on perfect word-level timestamps from ASR systems, which are notoriously noisy in practice. Because TurnGuide only requires the start time of a text chunk, it is significantly more robust to alignment errors.
- Prioritizing Text Generation Boosts Speech Quality: By adjusting the training loss weights to favor text tokens over speech tokens (e.g., a 3:1 ratio), the model is forced to focus more on semantic planning, which directly translates to higher-quality spoken outputs.
Key Results
TurnGuide was evaluated using GPT-4o to score the semantic meaningfulness and coherence of the generated dialogues (scaled from 0 to 10), alongside perplexity metrics and human absolute ratings.
- Superior Semantics: In unconditional generation, TurnGuide (with a 3:1 loss ratio) achieved a semantic score of 9.03, vastly outperforming the speech-only baseline (6.94) and the Moshi token-level training strategy (6.76).
- Lower Perplexity: Across multiple evaluation language models, TurnGuide consistently achieved lower perplexity, confirming its superior semantic modeling capabilities.

- Preserved Turn-Taking & Low Latency: TurnGuide maintained state-of-the-art performance on Full-Duplex-Bench for handling user interruptions and backchanneling. Crucially, it achieved a theoretical end-to-end interaction latency of 0.69 seconds, well within the acceptable threshold for real-time human conversation.
Limitations and Future Work
- Dataset Homogeneity: TurnGuide was primarily validated on the Fisher dataset. Because Fisher consists entirely of telephone conversations, it may not fully capture the stylistic diversity of other real-world conversational settings.
- Vocoder Constraints: While a 5-token chunk size is optimal for low latency, the underlying GLM-4-Voice vocoder requires processing at least 10 tokens together to maintain acoustic stability. Optimizing the vocoder for smaller chunk sizes remains an open area for future work.
Summary updated