Real-Time Short-Response Selection for Spoken Dialogue Systems

Hien Ohnaka, Kazuyo Onishi, Koichiro YoshinoarXiv 2026 · arXiv

TL;DR

This paper introduces a "fast-track" response system for spoken dialogue agents that masks the latency of Large Language Models (LLMs) and speech synthesis. By using contrastive learning to map dialogue context directly to a pool of pre-generated short responses (like "I see" or "Hmm..."), the system can start speaking in less than a millisecond after a user finishes their turn, buying time for the main response to be generated.

What Problem Does This Paper Address?

In modern spoken dialogue systems, the "cascaded" architecture—where audio is processed by Automatic Speech Recognition (ASR), then an LLM, and finally Text-to-Speech (TTS)—creates a significant bottleneck. Even if a system accurately predicts when it should speak (turn-taking), the computational overhead of generating a thoughtful response and synthesizing high-quality audio often results in an awkward silence of several seconds.

While some researchers try to speed up the models themselves, this often compromises quality. Others use simple "pre-prepared" fillers, but repeating the same "Hmm..." regardless of context makes the interaction feel robotic. The authors aim to bridge this gap by rapidly selecting contextually appropriate short responses that can be played immediately while the "slow" generation path works in the background.

Methods

The authors propose a real-time short-response selection task integrated into a Voice Activity Projection (VAP) model. The architecture is split into a "Fast system" and a "Slow system," as illustrated in the overview below:

Overview of the proposed real-time short-response selection.

Key Components:

  1. VAP Backbone: The system uses a VAP-based encoder that processes raw audio features (using CPC and HuBERT) to predict future speech activity. This allows the model to understand the "rhythm" and context of the conversation without waiting for ASR transcripts.
  2. Contrastive Learning: To select the best response, the model uses a contrastive objective (InfoNCE loss). It learns to project the current dialogue context and a set of candidate short responses into the same embedding space. At inference time, it simply picks the candidate with the highest cosine similarity to the current context.
  3. Multi-Head Prediction: Beyond selection, the model features a timing head to detect exactly when a turn starts and a classification head to decide if the event is a "backchannel" (a brief "uh-huh" while the user continues) or a "main response" (taking the floor).

The internal architecture of the VAP backbone is shown here: Original VAP architecture used to initialize the turn-taking backbone.

Original VAP architecture used to initialize the turn-taking backbone.

Datasets

The researchers used the Corpus of Everyday Japanese Conversation (CEJC), which contains natural, two-speaker dialogues.

  • Scale: Approximately 85 hours of audio across 247 sessions.
  • Processing: They extracted "short responses" from the first 2 seconds of main utterances and all backchannels to create a pool of candidates.
  • Partitioning: The data was split into training, validation, and test sets based on speaker IDs to ensure the model generalizes to new voices.
Total duration and numbers of sessions, unique speakers, and target responses in each partition.

Key Insights and Ideas

  • Independence from ASR: By operating on latent audio features rather than text, the selection process is immune to ASR errors and latency. This is crucial for "real-time" feel.
  • Masking vs. Speeding Up: The paper shifts the focus from making LLMs faster to making the perceived delay shorter. This "smoke and mirrors" approach is highly effective for human-robot interaction.
  • Order Independence: Unlike LLM-based selection methods, which can be biased toward the first few options in a list, the embedding-based approach evaluates all candidates (even up to 10,000) simultaneously and fairly.
  • Minimal Interference: A major "aha" moment was finding that adding the response selection task did not degrade the model's primary job: predicting turn-taking timing.

Key Results

  • Accuracy: The proposed method achieved a 15.72% Top-1 text accuracy, significantly outperforming both random chance (6.41%) and a baseline using the Gemma-4B LLM (7.55%).
  • Latency: On an Apple M5 CPU, the selection process took only 0.076 ms for a pool of 10,000 candidates. This is essentially instantaneous compared to the 100ms windows typically used in streaming audio processing.
  • Stability: The integration maintained high F1 scores for turn-timing prediction (around 89.7% at a 1.0s tolerance), proving that the multi-task learning was successful.
Text Top-1 accuracy (%) of short-response selection on 14,446 test events.

Limitations and Future Work

  • Consistency Issues: Currently, the "fast" filler and the "slow" main response are generated independently. If the system says "That's great!" (fast) but the LLM then generates a sad response (slow), the dialogue becomes incoherent. Future work aims to feed the selected filler back into the LLM as context.
  • One-to-Many Mapping: In natural speech, many different fillers might be appropriate for a single context (e.g., "I see," "Right," and "Uh-huh" are often interchangeable). The current model doesn't fully account for this diversity.
  • Acoustic Mismatch: The model is trained on real human fillers but uses synthetic TTS audio at inference. Bridging this "domain gap" in acoustic features could further improve selection accuracy.

Summary updated

Abstract

To realize spoken dialogue systems that can interact naturally with users, it is important for the system to respond to user utterances at appropriate timings. To achieve such natural turn-taking, the system needs not only to predict appropriate response timings but also to mask delays caused by language generation and speech synthesis so that they are not perceived by the user. We propose a real-time short-response selection task that can be integrated with existing turn-taking models to mask generation delay. To address this task, the proposed model learns the mapping between dialogue context and short responses using contrastive learning. When a turn-taking event is detected, the system selects a response from a set of pre-generated candidates. The selected response is played while the main response is being generated. Experimental results show that the proposed method learns to select short responses based on dialogue context, outperforming random and LLM-based baselines. Moreover, integrating response selection maintained turn-taking prediction performance and added minimal processing overhead.
BibTeX
@misc{ohnaka2026real,
  author = {Hien Ohnaka and Kazuyo Onishi and Koichiro Yoshino},
  title = {{Real-Time Short-Response Selection for Spoken Dialogue Systems}},
  year = {2026},
  eprint = {2611.09057},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09057},
}
Figure