Is Turn-Shift Distinguishable with Synchrony?

Jieyeon Woo, Yang Liu, Catherine Pelachaud, Catherine AchardLecture Notes in Computer Science 2023 · DOI · OpenAlex

Abstract

This paper explores whether interpersonal synchrony—the coordinated "dance" of conversation—can be utilized to automatically classify how participants swap speaking turns. By analyzing multimodal signals such as facial expressions, head movements, and pitch, the researchers developed a neural network model capable of distinguishing between smooth turn exchanges, interruptions, and backchannels. The study demonstrates that synchrony measures alone can achieve up to 75% accuracy, highlighting the relational nature of conversational dynamics.

Introduction

In human dialogue, the "speaking floor" is constantly negotiated. Handoffs can be "smooth exchanges," jarring "interruptions," or non-interruptive "backchannels" (e.g., "mm-hmm"). While previous research focused on individual behaviors like gaze or pitch to predict these moments, this study addresses a more relational question: Does the coordination between two people—their synchrony—change depending on the type of turn-shift? Understanding these patterns is essential for developing socially intelligent AI capable of navigating human dialogue naturally.

Methodology

The researchers developed a pipeline consisting of feature extraction, synchrony measurement, and classification:

  1. Segmentation: Turn-shifts were analyzed in three phases: Before (4s prior), During (4s window around the shift), and After (4s following).
  2. Multimodal Features: Data included facial Action Units (AUs), head motion (translation and rotation), and acoustics (pitch and loudness).
  3. Synchrony Measures: Three types of synchrony were calculated: Pearson Correlation (linear relationship), Time-lagged Cross-correlation (accounting for reaction time), and Dynamic Time Warping (DTW) (similarity between sequences of varying speeds).
  4. Classification: Models tested included Feedforward Neural Networks (FNN), 1D Convolutional Neural Networks (CNN), and Long Short-Term Memory (LSTM) networks.

Dataset

The study utilized the French portion of the NoXi database, featuring screen-mediated, face-to-face dyadic interactions. The dataset comprised 21 interactions (approx. 7.5 hours) with 1,403 smooth exchanges, 1,651 backchannels, and 929 interruptions, all manually annotated.

Key Insights and Ideas

The research established synchrony as a standalone signal for identifying social intent. A primary finding was the distinct "signature" of smooth turn exchanges, which exhibited a sudden increase in negative correlation for acoustic and head features. As one participant "fades out," the other "fades in," creating detectable opposite trends.

Pearson correlation coefficients for multimodal features across turn-shift types.

Furthermore, backchannels were found to be easily distinguishable from interruptions via time-lagged correlation in loudness and pitch. Listeners tend to synchronize their vocal energy with the speaker when providing brief feedback, a pattern captured by the model.

Time-lagged cross-correlation coefficients highlighting significant patterns in backchannels.

The study also emphasized the importance of the "After" phase. While predictive models are desirable, the behavior immediately following a shift provides the most informative signals for accurate classification.

Results

The top-performing model was a Feedforward Neural Network (FNN) utilizing all features across all three phases, achieving an accuracy of 0.75. Interestingly, a model using only statistically significant features performed worse (0.65 accuracy), suggesting that subtle, cross-modal interactions are vital for neural network performance even if they lack individual statistical significance.

When restricted to "Before" and "During" phases for real-time simulation, accuracy dropped to 0.53. This underscores the "wait and see" nature of social signals; the full context of an interlocutor's reaction is often necessary to categorize the turn-shift accurately.

Limitations and Future Work

The reliance on the "After" phase presents a challenge for real-time AI assistants, as a 4-second delay is impractical. Future research should aim to bridge this gap. Additionally, exploring cross-modal synchrony (e.g., syncing head nods with vocal pitch) and expanding the study to other cultures within the NoXi database (German and UK portions) could further refine the model's accuracy and generalizability.

Summary updated

BibTeX
@inproceedings{woo2023is,
  author = {Jieyeon Woo and Yang Liu and Catherine Pelachaud and Catherine Achard},
  title = {{Is Turn-Shift Distinguishable with Synchrony?}},
  booktitle = {Lecture Notes in Computer Science},
  year = {2023},
  doi = {10.1007/978-3-031-35894-4\_32},
}
Figure