Introduction
Social Relationship Recognition (SRR) aims to deduce the nature of relationships between individuals based on their interactions. While early research focused on static images, recent advancements have shifted toward video to capture complex spatio-temporal dynamics. However, existing video-based SRR methods often overlook three critical aspects of human interaction:
- The Symmetry Assumption: Most models assume relationships are mutual, ignoring the inherent asymmetry often present in human social roles and subjective experiences.
- Temporal Fragmentation: To reduce computational costs, many methods sample discrete, disconnected frames, which disrupts the continuous flow of real-world interactions.
- Ignoring Periodicity: Human interactions possess rhythmic patterns (e.g., question-answer cycles or synchronized laughter). Standard sequential models like RNNs or Transformers accumulate historical data but fail to explicitly model these long-term periodic behaviors.
Methodology
To address these limitations, we introduce AsyReC (Asymmetric Dyadic Relationship Classification), a multimodal framework that processes synchronized face, body, audio, and text features.
The framework is built upon three core innovations:
- Clip-Level Processing: Videos are segmented into continuous, synchronized multimodal clips to preserve the natural interaction flow.
- Node-Edge Attention Graph Network (NE-AGN): This "triplet graph" architecture handles relationship asymmetry. Two intra-person graphs use node attention to isolate individual multimodal cues, while an inter-person graph employs edge attention to model mutual behavioral influences.
- Periodic Temporal Signal Modeling: The temporal index of each clip is mapped onto 2π-periodic sine and cosine waves. This trigonometric encoding explicitly enables the network to identify rhythmic, repeating interaction patterns across the video duration.
Experimental Setup
AsyReC was evaluated on two primary public datasets:
- NoXi: Contains 168 videos of dyadic interactions. Crucially, this dataset features bidirectional labels—each participant rates the relationship from their own perspective (Stranger, Acquaintance, Friend, Very Good Friend)—making it ideal for testing asymmetric modeling.
- UDIVA: Comprises 1,160 videos with unidirectional labels (Known vs. Unknown), used to validate the model's generalizability at scale.
Results and Insights
AsyReC achieved state-of-the-art performance, with an Unweighted Average Recall (UAR) of 48.2% on NoXi and 59.4% on UDIVA, significantly outperforming baselines such as PGCN, LIReC, and CT.
Key findings from our analysis include:
- Modality Significance: Audio and text are highly effective for distinguishing "Strangers" from "Acquaintances." However, non-verbal cues (facial expressions and audio prosody) are critical for differentiating "Friends" from "Very Good Friends."
- Asymmetry Modeling: Ablation studies showed that while certain signals (like text) help predict one person's perspective, they may degrade the prediction for the other. Our graph-based fusion effectively resolves this cross-modal asymmetry.
- Temporal Dynamics: While early interactions suffice to identify "Strangers," distinguishing close friends requires mid-to-late video signals where more nuanced nonverbal exchanges occur.
- Superiority of Periodic Modeling: AsyReC showed a 44.1% absolute improvement in recognizing "Very good friends" compared to the Cumulative Transformer (CT) baseline, demonstrating that periodic modeling is more effective than simple memory accumulation for long-term interactions.
Conclusion
AsyReC demonstrates the power of modeling asymmetry and periodicity in social relationship recognition. While the model shows significant gains, it currently struggles with mixed-gender interactions due to dataset imbalances where same-gender pairs are overrepresented. Future work will focus on expanding dataset diversity to include more cross-cultural and mixed-gender pairs, as well as developing Explainable AI (XAI) techniques to better understand the specific social cues prioritized by the network.
Summary updated