TL;DR
This paper introduces a novel neural pipeline that transforms isolated, unsynchronized voice recordings into natural, dyadic conversations. By decoupling conversational timing from speech content, the system generates realistic listener "back-channels" (like "uh-huh" or "wow") and manages complex turn-taking dynamics with fine-grained animator control.
What Problem Does This Paper Address?
In the video game and animation industries, dialogue is rarely recorded as a live conversation. Instead, actors record their lines in isolation, often without hearing their scene partner. When these "voice-lines" are simply spliced together, the result sounds robotic—lacking the natural overlaps, varied pauses, and subtle listener interjections (back-channels) that characterize human interaction.
While modern Large Language Models (LLMs) can generate speech, they often struggle with "full-duplex" dynamics—the ability to listen and speak simultaneously. Existing models either produce incoherent speech when forced to handle two channels or fail to provide the precise timing control required by professional animators. This paper seeks to bridge that gap by creating a system that takes fixed audio clips and "schedules" them into a believable, dynamic exchange.
Methods
The authors propose a two-stage pipeline:
- Back-channel Predictor: This is a multimodal classifier (using RoBERTa for text and HuBERT for audio) that analyzes a speaker's turn to decide if the listener should interject. It classifies opportunities into three categories: No back-channel, Acknowledgment (low-salience, e.g., "yeah"), or Reactive (high-salience, e.g., "really?!").
- Neural Scheduler: This component determines the precise start times for every clip. The authors formulate scheduling as a "next-token prediction" task using a full-duplex LLM (based on GLM-4-Voice).
A key technical innovation is the Duration Prediction Head. Standard training for these models suffers from "inertia bias," where the model simply learns to repeat the current state (silence or speech) because transitions are statistically rare. To fix this, the authors add an auxiliary task: at every frame, the model must predict how many frames of silence remain before the next speech segment begins.
At inference time, the scheduler doesn't just generate tokens; it acts as a scoring function. It searches through candidate start times and chooses the arrangement that the model deems most "natural" (lowest Negative Log-Likelihood).
Datasets
- Seamless Interaction Dataset: A large-scale collection of two-channel conversations used for training both the predictor and the scheduler.
- Fisher Corpus: Used exclusively for evaluation to ensure the model generalizes to unseen conversational data.
- Full-Duplex Bench: A specialized benchmark used to evaluate back-channel behavior (frequency, timing, and intrusion).
Key Insights and Ideas
- Decoupling is Key: By separating what is said (the fixed audio clips) from when it is said, the system avoids the "intelligibility degradation" common in generative audio models. The timing becomes the sole degree of freedom for optimization.
- Audio Trumps History: In back-channel prediction, the authors found that the current speaker's audio and immediate future text are far more predictive than the long-term conversational history.
- Solving Inertia Bias: The "aha moment" for conversational timing was realizing that a standard next-token loss isn't enough. Forcing the model to "look ahead" via the duration prediction head was essential to prevent the timing from collapsing into a single, repetitive spike.
Key Results
- Turn-Taking Accuracy: The Neural Scheduler produced timing statistics (pauses, gaps, and overlaps) significantly closer to human ground truth than existing state-of-the-art models like Moshi or dGSLM.
- Back-channel Quality: On the Full-Duplex Bench, the system achieved a "Turn-Override Rate" of 0.000 (meaning it never awkwardly cut off the main speaker) and the highest back-channel frequency (0.062/sec).
- Human Preference: In a user study, participants significantly preferred the system's output over baselines. For dramatic, expressive performances, the preference for the neural scheduler was particularly strong (64.3%), as it captured the nuances of emotional timing that simple heuristics miss.
Limitations and Future Work
The current model operates on fixed voice-line clips. This means it cannot automatically introduce natural speech disfluencies, such as stutters, repetitions, or mid-sentence hesitations, which are common in real-world "messy" conversations.
Additionally, while the system decides when to place a back-channel, the content (the specific word used) is currently selected from a pre-defined vocabulary provided by the animator. A promising future direction is a hybrid approach where an LLM selects semantically appropriate back-channel words while the Neural Scheduler handles the precise, high-resolution timing.
Summary updated