Turn-Taking Prediction Based on Detection of Transition Relevance Place

Kohei Hara, Koji Inoue, Katsuya Takanashi, Tatsuya KawaharaInterspeech 2019

TL;DR

This paper proposes a novel two-step approach to predict when a conversational partner will take the speaking turn, by first identifying "transition relevance places" (TRPs) – moments where a turn could end – and then predicting the actual turn-taking event. This method significantly improves turn-taking prediction accuracy compared to traditional approaches, especially in natural, multi-utterance dialogues.

What Problem Does This Paper Address?

In human-robot and human-agent conversations, smooth turn-taking is crucial for natural interaction and user satisfaction. Unlike simple query-response systems that rely on fixed silence durations, more complex dialogues involve users speaking multiple utterances per turn. Predicting when to take the conversational floor in these scenarios is challenging because turn-taking decisions can be subjective and context-dependent. Existing methods often directly predict the "take" or "not take" decision, but struggle with the inherent ambiguity and the imbalance of "take" events in natural conversations. This paper aims to improve turn-taking prediction by explicitly modeling the intermediate decision of whether a potential turn-ending point (a TRP) has been reached.

Methods

The core of the proposed method is a two-step process:

  1. TRP Detection: This model identifies potential moments where a speaker could yield the floor. It uses a hierarchical LSTM architecture that processes prosodic features (power, F0), speech features (log Mel filterbank), and linguistic features (word2vec). The model is trained on a binary classification task: TRP or not TRP.
  2. Turn-Taking Prediction at TRP: Once a TRP is detected, this second model predicts whether the listener will actually take the turn ("take") or continue listening ("not take"). This model uses a similar LSTM architecture and feature set but is trained only on segments identified as TRPs.

The final turn-taking decision is made by combining the probabilities from both models. If the probability of a TRP occurring and the listener taking the turn is above a threshold (0.5), the system predicts a "take"; otherwise, it predicts "not take." This separation allows for specialized models that can better handle the distinct decision-making processes involved.

Datasets

The authors used a newly annotated human-robot dialogue corpus. This corpus consists of three dialogue tasks:

  • Job interview task: Simulated interviews where ERICA (the robot) is the interviewer.
  • Attentive listening task: Subjects talk about a theme, and ERICA provides listening cues.
  • Speed dating task: Casual first-time meetings between subjects and ERICA.

The corpus was annotated for Inter-pausal Units (IPUs), Long Utterance Units (LUUs), and Dialogue Acts (DAs). Crucially, the authors conducted manual annotation of Transition Relevance Places (TRPs) on this corpus, ensuring objectivity through inter-annotator agreement measures (Cohen's kappa scores ranging from 0.784 to 0.817).

Key Insights and Ideas

  • Decomposing Turn-Taking: The paper's central idea is that turn-taking isn't a single decision but a two-stage process: first, identifying if a turn can end (TRP), and second, deciding whether to end it. This decomposition simplifies the problem for each model.
  • TRP as a Crucial Intermediate Step: The analysis shows a strong correlation between TRPs and actual turn-taking. When a TRP is identified, the likelihood of a turn-switch ("take") increases significantly compared to non-TRP segments.
  • Handling Imbalanced Data: By training the turn-taking prediction model only on TRP segments, the authors effectively address the class imbalance issue where "take" events are much rarer than "not take" events in the overall dataset. This leads to improved recall for the "take" class.
  • Objective TRP Annotation: The effort to ensure objective TRP annotation through clear guidelines and inter-annotator agreement is vital for building a reliable TRP detection model.

Key Results

  • The proposed two-step model significantly improved turn-taking prediction accuracy across all dialogue tasks compared to a baseline model that directly predicted turn-taking.
  • Specifically, the F-score for turn-switch prediction was notably enhanced, with drastic improvements in recall rates. For example, in the speed dating task, the proposed model achieved an F-score of 79.3, compared to the baseline's 68.9.
  • The TRP detection model achieved high accuracy, exceeding 80% for all dialogue tasks, demonstrating its effectiveness in identifying potential turn-ending points.
  • When evaluating the turn-taking prediction model only on TRP segments (Table 5), the proposed method also outperformed the baseline, particularly in recall.

Limitations and Future Work

  • Exceptional Cases: The paper acknowledges that there are "exceptional cases" where a turn-switch might occur even if a segment is not annotated as TRP, or vice-versa. The current model assumes "not take" when a place is not TRP, which might miss these exceptions (e.g., interruptions or barge-ins).
  • Future Work: The authors suggest that incorporating more sophisticated models for TRP detection and turn-taking prediction, potentially exploring end-to-end learning or more advanced feature representations, could further improve performance. Addressing the exceptional cases of interruptions is also a potential avenue for future research.

Summary updated

BibTeX
@inproceedings{hara2019turn,
  author = {Kohei Hara and Koji Inoue and Katsuya Takanashi and Tatsuya Kawahara},
  title = {{Turn-Taking Prediction Based on Detection of Transition Relevance Place}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure