Analyzing Temporal Dynamics of Dyadic Synchrony in Affective Interactions

Zhaojun Yang, Shrikanth S. NarayananInterspeech 2016 · DOI

Introduction

In human conversation, we often unconsciously mirror our partner's gestures, speech patterns, and emotions—a phenomenon known as dyadic synchrony. While traditionally measured as a static average over an entire interaction, this research, presented at INTERSPEECH 2016, posits that synchrony is a fluid, time-varying process. By treating synchrony as a dynamic "dance," the authors explore how its temporal fluctuations correlate with the emotional climate of a conversation.

Methodology: Quantifying the "Dance"

The core challenge in modeling synchrony is capturing turn-by-turn coordination without being skewed by individual speaking styles or quirks. The researchers developed a robust statistical pipeline to address this:

  1. Fisher Vector Representation: Each turn is encoded as a Fisher Vector, which measures how a specific behavior deviates from a global model of all behaviors (a Gaussian Mixture Model). This normalization ensures the metric captures interpersonal alignment rather than individual traits.
  2. Synchrony Metric: Synchrony ($\sigma$) is defined as the squared cosine similarity between the Fisher Vectors of adjacent turns from two interlocutors. This approach allows for a granular, turn-by-turn assessment of coordination.
Turn-wise interaction model for calculating synchrony between Interlocutor A and B.
  1. Functional Data Analysis (FDA): To analyze the temporal "shape" of synchrony, the sequence of turn-wise scores is converted into a continuous mathematical function using B-splines. Functional Principal Component Analysis (FPCA) is then used to identify dominant patterns of variability over time, while Functional Canonical Correlation Analysis (FCCA) measures the coupling between synchrony and emotional curves.

Data and Features

The study utilized the USC CreativeIT database, containing 50 dyadic theatrical improvisations. The researchers extracted multimodal features to capture both vocal and physical dynamics:

  • Acoustic: Pitch, Energy, and MFCCs from high-quality audio recordings.
  • Gestural: Head and hand joint angles derived from 45 full-body motion capture markers. The data was annotated with both global emotional ratings and time-continuous "Feeltrace" annotations for activation (calm vs. excited) and valence (positive vs. negative).

Key Results and Insights

The analysis revealed distinct temporal signatures for different modalities and emotional states. The proposed synchrony measure was first validated by successfully distinguishing between real interactions and "shuffled" random turn pairs ($p < 0.001$), proving it captures genuine interpersonal dynamics.

Acoustic vs. Gestural Stability

A significant finding was the difference in stability between modalities. Vocal synchrony tends to remain relatively stable throughout an interaction. In contrast, gestural synchrony is much more oscillatory, characterized by frequent peaks and troughs of coordination.

Functional PCA results showing the mean curve and variability for Gesture and Acoustic synchrony over time.

Emotional Coupling

The "shape" of these synchrony curves is tightly linked to emotion:

  • Valence (Positive/Negative): High-valence (positive) interactions exhibited significantly higher synchrony. Moreover, the way synchrony fluctuates (the 2nd variation component) differed markedly between positive and negative dyads. The researchers found a high degree of shared variance between valence and gesture synchrony ($r^2 = 0.719$).
  • Activation (Calm/Excited): Calm dyads (low activation) were generally more synchronized than excited or stressed ones. FCCA results showed that activation and synchrony often move in "antiphase"—as one increases, the other tends to decrease.

Conclusion

This work demonstrates that the temporal dynamics of synchrony are a rich source of information about the emotional state of a dyad. By moving beyond static averages to functional representations, the researchers have provided a framework for more nuanced affective computing systems. Future work aims to integrate these temporal dynamics into automatic emotion recognition systems, looking not just at what a person is doing, but at how well they are aligned with their partner over time.

Summary updated

BibTeX
@inproceedings{yang2016analyzing,
  author = {Zhaojun Yang and Shrikanth S. Narayanan},
  title = {{Analyzing Temporal Dynamics of Dyadic Synchrony in Affective Interactions}},
  booktitle = {Interspeech},
  year = {2016},
  doi = {10.21437/interspeech.2016-158},
}
Figure