Neural Generation of Dialogue Response Timings

Matthew Roddy, Naomi HarteACL 2020 · DOI

Introduction

Current spoken dialogue systems (SDSs) typically aim to minimize response latency, yet human conversation is characterized by nuanced timing that varies based on content and acoustic cues. This paper introduces RTNet, a neural architecture designed to predict naturalistic response offsets. By encoding upcoming system responses and monitoring user speech in real-time, RTNet—and its Variational Autoencoder (VAE) extension—enables systems to generate timings that feel significantly more human.

The Challenge of Naturalistic Turn-Taking

In standard SDSs like Alexa or Siri, the goal is often to minimize the gap without interrupting. However, humans use timing as a communicative signal: "backchannels" (e.g., "uh-huh") often overlap with the speaker, while "dispreferred responses" (e.g., declining an invitation) are typically preceded by a noticeable delay.

Existing systems often ignore this context, leading to robotic interactions. The challenge lies in building a model that understands how response content dictates timing while maintaining the ability to process information incrementally in real-time.

The RTNet Architecture

The authors propose two primary models to address these challenges:

  1. RTNet (Response Timing Network): This model consists of an Encoder and an Inference Network. The Encoder uses three stacked Bi-LSTMs to fuse linguistic features (GloVe embeddings) and acoustic features (pitch, intensity, etc.) of the upcoming system response into a single vector ($h_z$). The Inference Network is a frame-by-frame LSTM that monitors the user’s current speech and uses $h_z$ to predict the probability of starting a turn at 50ms intervals.
  2. RTNet-VAE: To support incremental dialogue where a full response might not be ready, the authors integrated a Variational Autoencoder. This maps response types into a continuous latent space, allowing the Dialogue Manager to sample a point (e.g., a "backchannel" region) to derive appropriate timing without needing a complete sentence encoding.
High-level architecture of the RTNet model for real-time response timing prediction.

Data and Evaluation

The models were trained and evaluated on the Switchboard-1 Release 2 corpus, comprising over 2,400 dyadic telephone conversations (approx. 260 hours). The authors extracted turn pairs and cross-referenced them with NXT-format annotations and the Switchboard Dialog Act Corpus (SWDA) to analyze performance across different dialogue acts, such as statements versus yes/no questions.

Performance and Insights

RTNet successfully learned that different Dialogue Acts require distinct offsets. For example, it accurately predicted that "no" responses should be more delayed than "yes" responses. Ablation studies revealed that while acoustic features help the model anticipate the end of a user's phrase, linguistic features are crucial for determining the overall timing distribution mode.

Key Results

  • Quantitative: The Full RTNet model achieved a Mean Absolute Error (MAE) of 0.4539 seconds, significantly outperforming a "Fixed Probability" baseline (1.4546s) that simply guesses the average human gap.
  • Qualitative: In A/B human preference tests, listeners found the model's varied timings more realistic than a fixed modal response, particularly when the model provided context-specific delays or overlaps.
  • VAE Efficiency: The RTNet-VAE performed nearly as well as the full encoder model (MAE ~0.46s) while offering higher computational efficiency for real-time deployment.
Comparison of response offset distributions between the Full Model and a Fixed Probability baseline.

Conclusion and Future Work

RTNet demonstrates that response timing can be modeled as a function of dialogue content and real-time acoustic monitoring. However, limitations remain: the model assumes a Dialogue Manager has already decided what to say, and it does not yet handle the logic of whether to remain silent. Future work could extend this framework to model internal timings within a response, such as mid-sentence pauses for emphasis, and further refine the system's robustness to real-world ASR latency.

Summary updated

BibTeX
@inproceedings{roddy2020neural,
  author = {Matthew Roddy and Naomi Harte},
  title = {{Neural Generation of Dialogue Response Timings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.221},
}
Figure