Modeling social interaction dynamics using temporal graph networks

Joanne Taery Kim, Archit Naik, Isuru Jayarathne, Sehoon Ha, Jouh Yeong Chew

Abstract

As robots are increasingly integrated into social environments, they must navigate the complexities of multi-party human interactions. This paper proposes a framework using Temporal Graph Networks (TGNs) to model group dynamics by representing participants as nodes and their gaze as evolving edges. By employing a two-phase learning approach, the model predicts future gaze and speaking turns. Experimental results on the FUMI-MPF dataset demonstrate that TGNs significantly outperform traditional baselines. Notably, the study finds that simple one-hot encoding is more efficient and effective than complex BERT embeddings for real-time social modeling, reducing data complexity by over 50x.

Introduction

Human-robot interaction (HRI) has traditionally focused on dyadic (one-on-one) scenarios. However, real-world social environments are typically multi-party, involving intricate non-verbal cues like gaze, seating arrangements, and social roles. Existing models often fail to capture the relational and temporal dependencies inherent in these interactions. This research addresses these challenges by framing social dynamics as a link prediction problem on a dynamic graph.

Methodology: Social Interaction as a Temporal Graph

The core innovation lies in representing a social group as a graph where each person is a node. A directed edge from node A to node B indicates that person A is looking at person B. Because social interactions are fluid, these edges change over time, creating a temporal graph.

Conceptual overview of the social interaction model and the Temporal Graph Network (TGN) architecture.

Temporal Graph Networks (TGN)

The researchers utilized TGNs to maintain a "memory" for each node, which is updated as interactions occur. This allows the model to capture the flow of conversation and long-term dependencies over time, making it more flexible than models that expect a fixed number of inputs.

Two-Phase Learning

  1. Phase A (Next Gaze Prediction): The model is first trained using self-supervised learning to predict future gaze interactions (links). This builds a "social backbone" for the model.
  2. Phase B (Next Speaker Prediction): The features learned in Phase A are used to fine-tune the model to predict which node (person) will speak next.

Message Encoding: BERT vs. One-hot

The study compared two encoding methods for social cues:

  • BERT Encoding: High-dimensional (768 elements) embeddings from a pre-trained language model.
  • One-hot Encoding: A low-dimensional (14 elements) binary vector representing states.

Experimental Setup and Dataset

The model was evaluated using the FUMI-MPF dataset, which includes 24 group sessions with 110 participants. Data was collected using multi-modal sensors, including Tobii eye-tracking glasses and voice activity detectors. Group sizes varied from 3 to 6 individuals in a music appreciation discussion context facilitated by different leaders.

Results and Discussion

The TGN approach demonstrated substantial improvements over a "History Baseline." For next-gaze prediction, the model achieved a 37.0% improvement in F1-score and a 24.2% boost in accuracy. For next-speaker prediction, it saw a 29.0% improvement in F1-score.

Comparison of various TGN model configurations based on test average precision and computational time per epoch.

Efficiency and Performance

A key finding was that One-hot encoding matched or exceeded the performance of BERT encoding while being significantly more computationally efficient. By reducing the message size from 768 to 14 elements, the model becomes much more suitable for real-time deployment on robotic platforms. Figure 6 illustrates the trade-offs between precision and training time for various TGN configurations, highlighting that models like tgn-attn and tgn-mean provide high precision with manageable computational costs compared to more complex variants like tgn-2l.

Conclusion and Future Work

By treating social interactions as dynamic graphs, this work provides a robust foundation for robots to understand group dynamics. While predicting the next speaker remains a challenging task, the use of gaze-based pre-training significantly enhances performance. Future research will explore the integration of additional modalities, such as body language, vocal pitch, and speech rate, to further refine the model's social intuition in diverse scenarios.

Citation: J. Taery Kim, Archit Naik, Isuru Jayarathne, Sehoon Ha, Jouh Yeong Chew. "Modeling social interaction dynamics using temporal graph networks." arXiv:2404.06611v1 [cs.HC] 5 Apr 2024.

Summary updated

BibTeX
@inproceedings{kim2024modeling,
  author = {Joanne Taery Kim and Archit Naik and Isuru Jayarathne and Sehoon Ha and Jouh Yeong Chew},
  title = {{Modeling social interaction dynamics using temporal graph networks}},
  booktitle = {RO-MAN},
  year = {2024},
  doi = {10.1109/ro-man60168.2024.10731450},
}
Figure