Abstract
Full-Duplex Speech Language Models (FD-SLMs) are specialized foundation models designed to enable natural, real-time spoken interactions by modeling complex conversational turn-taking such as interruptions, backchannels, and overlapping speech. End-to-end (e2e) FD-SLMs leverage real-world double-channel conversational data to capture nuanced two-speaker dialogue patterns for human-like interactions, but their conversational abilities often degrade compared to pure-text conversation due to prolonged speech sequences and limited high-quality spoken dialogue data. Although interleaved text-speech generation could mitigate this degradation, integrating discrete text tokens into continuous double-channel audio streams could disrupt the precise time alignment required for fluid interaction. To address this, we propose TurnGuide, a novel text-speech interleaved generation approach for e2e FD-SLMs that dynamically segments assistant speech into dialogue turns and interleaves turn-level text and speech generation. This approach allows FD-SLMs to integrate the semantic intelligence of LLMs without compromising the natural acoustic flow. Extensive experiments show that TurnGuide not only significantly improves e2e FD-SLMs to produce semantically meaningful, coherent speech but also achieves state-of-the-art performance on various turn-taking events.
Introduction
Full-Duplex Speech Language Models (FD-SLMs) represent a significant milestone in voice AI, enabling natural, two-way conversations where models can listen and speak simultaneously, handle interruptions, and provide backchannels. However, current end-to-end (e2e) models often suffer from a "semantic gap." While they excel at capturing the acoustic rhythm of speech, they frequently struggle with semantic coherence, producing responses that are less meaningful than those of text-based Large Language Models (LLMs).
Previous attempts to bridge this gap by interleaving text tokens into the audio stream (e.g., the Moshi model) often relied on token-level interleaving. This approach fragments semantic context, preventing the model from "planning" full sentences and disrupting the precise synchronization necessary for fluid interaction.
Methodology: Dynamic Turn-Level Interleaving
TurnGuide addresses these challenges through Dynamic Turn-Level Interleaving, a framework that segments conversations into natural dialogue turns to provide coherent text guidance.
The methodology is divided into two primary stages:
- Dynamic Turn Segmentation & Alignment: The system utilizes Voice Activity Detection (VAD) to identify speech segments, which are then merged into Inter-Pausal Units (IPUs). Using Whisper ASR with word-level timestamps, the conversation transcript is aligned to these IPUs, ensuring text guidance is synchronized with the start of each turn.
- Text-Guided Full-Duplex Modeling: Built on a GLM-4-Voice backbone, the model employs channel-wise interleaving to maintain two-speaker context. Within the assistant's channel, it generates a small chunk of text (e.g., 5 tokens) immediately followed by the corresponding audio (400ms), providing a "semantic roadmap" for speech generation.
Experimental Setup
The model was trained and evaluated using the Fisher Dataset, a corpus of 2,000 hours of double-channel English telephone conversations. To evaluate fine-grained interaction behaviors, the researchers utilized Full-Duplex-Bench, which measures performance in pause handling, backchanneling, turn-taking, and interruptions.
Key Results
TurnGuide demonstrates significant improvements in both semantic coherence and interaction quality.
- Semantic Superiority: In GPT-4o-based evaluations of meaningfulness, TurnGuide achieved an overall score of 7.79, significantly outperforming the Moshi-style token-alignment baseline (5.58) and speech-only models (4.76).
- Human-Centric Performance: Human listening tests confirmed these findings, with TurnGuide reaching a Meaningfulness Mean Opinion Score (M-MOS) of 3.26, nearing the ground truth of 4.25.
- Interaction Excellence: On the Full-Duplex-Bench, TurnGuide achieved state-of-the-art results in pause handling and user interruption latency, demonstrating superior responsiveness compared to existing cascaded or e2e systems.
Discussion and Future Work
The success of TurnGuide highlights that text guidance is most effective when it carries sufficient context. By grouping text into turns, the model can effectively plan its responses. Furthermore, the timing of text insertion is critical; turn-level alignment provides the optimal balance between context availability and real-time synchronization.
Future research will focus on expanding the model's capabilities to include diverse spontaneous dialogue styles beyond telephone calls and integrating multi-modal inputs like video or gestures. Additionally, implementing safety-tuning frameworks (e.g., RLHF) will be essential for addressing potential biases inherited from raw conversational data.
Demos are available at https://dreamtheater123.github.io/TurnGuide-Demo/.
Summary updated