AsyReC: A Multimodal Graph-Based Framework for Spatio-Temporal Asymmetric Dyadic Relationship Classification

Wang Tang, Fethiye Irmak Doğan, Linbo Qing, Hatice Güneş

Introduction

Social Relationship Recognition (SRR) aims to deduce the nature of relationships between individuals based on their interactions. While early research focused on static images, recent advancements have shifted toward video to capture complex spatio-temporal dynamics. However, existing video-based SRR methods often overlook three critical aspects of human interaction:

  1. The Symmetry Assumption: Most models assume relationships are mutual, ignoring the inherent asymmetry often present in human social roles and subjective experiences.
  2. Temporal Fragmentation: To reduce computational costs, many methods sample discrete, disconnected frames, which disrupts the continuous flow of real-world interactions.
  3. Ignoring Periodicity: Human interactions possess rhythmic patterns (e.g., question-answer cycles or synchronized laughter). Standard sequential models like RNNs or Transformers accumulate historical data but fail to explicitly model these long-term periodic behaviors.

Methodology

To address these limitations, we introduce AsyReC (Asymmetric Dyadic Relationship Classification), a multimodal framework that processes synchronized face, body, audio, and text features.

The AsyReC framework architecture, highlighting multimodal feature extraction, graph-based interaction modeling, and periodic temporal signal modeling.

The framework is built upon three core innovations:

  • Clip-Level Processing: Videos are segmented into continuous, synchronized multimodal clips to preserve the natural interaction flow.
  • Node-Edge Attention Graph Network (NE-AGN): This "triplet graph" architecture handles relationship asymmetry. Two intra-person graphs use node attention to isolate individual multimodal cues, while an inter-person graph employs edge attention to model mutual behavioral influences.
  • Periodic Temporal Signal Modeling: The temporal index of each clip is mapped onto 2π-periodic sine and cosine waves. This trigonometric encoding explicitly enables the network to identify rhythmic, repeating interaction patterns across the video duration.

Experimental Setup

AsyReC was evaluated on two primary public datasets:

  • NoXi: Contains 168 videos of dyadic interactions. Crucially, this dataset features bidirectional labels—each participant rates the relationship from their own perspective (Stranger, Acquaintance, Friend, Very Good Friend)—making it ideal for testing asymmetric modeling.
  • UDIVA: Comprises 1,160 videos with unidirectional labels (Known vs. Unknown), used to validate the model's generalizability at scale.

Results and Insights

AsyReC achieved state-of-the-art performance, with an Unweighted Average Recall (UAR) of 48.2% on NoXi and 59.4% on UDIVA, significantly outperforming baselines such as PGCN, LIReC, and CT.

Confusion matrices comparing AsyReC against baseline models (PGCN, LIReC, CT) on the NoXi dataset.

Key findings from our analysis include:

  • Modality Significance: Audio and text are highly effective for distinguishing "Strangers" from "Acquaintances." However, non-verbal cues (facial expressions and audio prosody) are critical for differentiating "Friends" from "Very Good Friends."
  • Asymmetry Modeling: Ablation studies showed that while certain signals (like text) help predict one person's perspective, they may degrade the prediction for the other. Our graph-based fusion effectively resolves this cross-modal asymmetry.
  • Temporal Dynamics: While early interactions suffice to identify "Strangers," distinguishing close friends requires mid-to-late video signals where more nuanced nonverbal exchanges occur.
  • Superiority of Periodic Modeling: AsyReC showed a 44.1% absolute improvement in recognizing "Very good friends" compared to the Cumulative Transformer (CT) baseline, demonstrating that periodic modeling is more effective than simple memory accumulation for long-term interactions.

Conclusion

AsyReC demonstrates the power of modeling asymmetry and periodicity in social relationship recognition. While the model shows significant gains, it currently struggles with mixed-gender interactions due to dataset imbalances where same-gender pairs are overrepresented. Future work will focus on expanding dataset diversity to include more cross-cultural and mixed-gender pairs, as well as developing Explainable AI (XAI) techniques to better understand the specific social cues prioritized by the network.

Summary updated

Abstract

Dyadic social relationships, which refer to relationships between two individuals who know each other through repeated interactions (or not), are shaped by shared spatial and temporal experiences. Current computational methods for modeling these relationships face three major challenges: (1) the failure to model asymmetric relationships, e.g., one individual may perceive the other as a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">friend</i> while the other perceives them as an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">acquaintance</i>, (2) the disruption of continuous interactions by discrete frame sampling, which segments the temporal continuity of interaction in real-world scenarios, and (3) the limitation to consider periodic behavioral cues, such as rhythmic vocalizations or recurrent gestures, which are crucial for inferring the evolution of dyadic relationships. To address these challenges, we propose AsyReC, a multimodal graph-based framework for asymmetric dyadic relationship classification, with three core innovations: (i) a triplet graph neural network with node-edge dual attention that dynamically weights multimodal cues to capture interaction asymmetries (addressing challenge 1); (ii) a clip-level relationship learning architecture that preserves temporal continuity, enabling fine-grained modeling of real-world interaction dynamics (addressing challenge 2); and (iii) a periodic temporal encoder that projects time indices onto sine/cosine waveforms to model recurrent behavioral patterns (addressing challenge 3). Extensive experiments on two public datasets demonstrate state-of-the-art performance, while ablation studies validate the critical role of asymmetric interaction modelling and periodic temporal encoding in improving the robustness of dyadic relationship classification in real-world scenarios. Our code is publicly available at: https://github.com/tw-repository/AsyReC.
BibTeX
@article{tang2025asyrec,
  author = {Wang Tang and Fethiye Irmak Doğan and Linbo Qing and Hatice Güneş},
  title = {{AsyReC: A Multimodal Graph-Based Framework for Spatio-Temporal Asymmetric Dyadic Relationship Classification}},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2025},
  doi = {10.1109/tcsvt.2025.3616347},
}

Loading related content...

Figure