Social Network Extraction and Analysis Based on Multimodal Dyadic Interaction

Sergio Escalera, Xavier Baró, Jordi Vitrià, Petia Radeva, Bogdan Raducanu

TL;DR

This paper introduces a novel framework for automatically extracting and analyzing social networks from multimodal dyadic interactions (audio and video). By fusing audio-visual cues to accurately segment speakers and then employing an "Influence Model," they quantify interpersonal influence, creating a directed graph representing the social network. This approach offers a way to understand social dynamics in unscripted, real-world conversations.

What Problem Does This Paper Address?

Understanding social interactions is crucial, and Social Network Analysis (SNA) is a powerful tool for modeling these relationships. However, most existing methods rely on explicit user input (like "likes" or "friends" on social media) or analyze structured interactions. This paper tackles the challenge of analyzing social interactions from implicit, multimodal cues in unscripted, face-to-face conversations. Specifically, it aims to move beyond simply identifying who is speaking to understanding how individuals influence each other within these natural dialogues, which is difficult to capture with traditional methods or even human annotation due to its subtle nature.

Methods

The framework consists of three main stages:

  1. Audio-Visual Feature Extraction and Fusion:
    • Audio: Mel-Frequency Cepstral Coefficients (MFCCs), energy, and their derivatives are extracted. Speaker segmentation is achieved using a Generalized Likelihood Ratio for coarse segmentation, followed by hierarchical clustering and Viterbi decoding.
  2. Visual: Face detection (Viola-Jones) and mouth region extraction are performed. Histogram of Oriented Gradients (HOG) are used to extract features from the mouth region. A one-class classification approach using Dynamic Time Warping (DTW) is employed to distinguish speaking from non-speaking patterns.
  3. Fusion: Stacked Sequential Learning (SSL) is used to combine audio and visual features. This technique leverages predicted labels from neighboring samples as new features for a second learning stage, effectively incorporating temporal information and improving the robustness of speaker segmentation.
  4. Social Network Extraction (Influence Model):
    • The "Influence Model" (InfModel) is employed to model interactions between participants. This model, based on a coupled Hidden Markov Model (HMM), quantifies the "influence" one person has over another. The states of the model encode the integrated audio-visual data, and the transition probabilities between states are parameterized by "influence" coefficients ($a_{ij}$). These coefficients represent the constant factors determining how much a person's speaking state at time t depends on another person's state at time t-1.
  5. Social Network Analysis:
    • The extracted influence relationships form a directed graph where link weights are the influence coefficients.
  6. Standard SNA centrality measures are computed:
  7. Degree Centrality: In-degree (receiving influence) and Out-degree (exerting influence).
  8. Closeness Centrality: Measures how easily a person can reach others.
  9. Betweenness Centrality: Identifies individuals who bridge different parts of the network.
  10. Eigenvector Centrality: Measures influence based on the influence of one's neighbors.

Datasets

The study uses a subset of videos from the publicly available New York Times' Blogging Heads opinion blog [21]. This dataset consists of dyadic conversations where two people discuss a topic. The authors collected 17 videos from 15 different people, selecting those who were most active in the blog to ensure a representative structure of the active participants.

Key Insights and Ideas

  • Implicit Social Dynamics: The core idea is that subtle, nonverbal cues in natural conversations carry significant information about social influence and relationships, which can be quantified.
  • Multimodal Fusion for Robustness: Combining audio and visual information significantly improves speaker segmentation accuracy compared to using either modality alone, which is critical for accurate influence modeling.
  • Influence as a Directed Graph: Representing social networks as directed graphs with influence weights ($a_{ij}$) provides a nuanced view of relationships, moving beyond simple connections to understand the direction and strength of interpersonal impact.
  • Centrality Reveals Roles: Different centrality measures effectively highlight distinct roles within the network, such as the most influential person (high out-degree, eigenvector centrality), a crucial connector (high betweenness), or a passive recipient (high in-degree).
  • Proof-of-Concept for Real-World Interactions: The framework is designed to work with unscripted, informal interactions, making it applicable to real-life scenarios without imposing constraints on participants.

Key Results

  • Audio-Visual Fusion Performance: The audio-visual fusion methodology significantly improved speaking/non-speaking classification accuracy for most subjects (28 out of 34), with improvements often exceeding 10% compared to individual modalities. For example, in conversation 1-2, A-V fusion achieved 81.90% for the left speaker and 78.87% for the right, compared to lower scores for video or audio alone.
  • Statistical Significance: Statistical tests (Friedman and Nemenyi) confirmed that the proposed audio-visual fusion methodology achieved statistically significant improvements over using audio or video alone.
  • Network Structure and Centrality: The analysis of the extracted social network revealed distinct roles. For instance, Node 1 was identified as the most important and influential person, influencing up to four others. Node 4 acted as a crucial cohesion factor (hub) with high betweenness centrality, bridging three subgroups. Node 8 was found to be the least relevant, with minimal impact on the network.

Limitations and Future Work

  • Lack of Absolute Ground Truth: The authors acknowledge that for informal conversations, establishing an absolute ground truth for interactional processes (especially influence) is nearly impossible, even for human annotators. Their approach relies solely on non-verbal analysis, abstracting away from conversation content.
  • Limited Dataset Size: While the authors believe the dataset size did not significantly influence conclusions, it was a subset of 17 videos from 15 people.
  • Future Work:
    • Extend the approach to larger-scale social interactions and different scenarios.
  • Study long-term interactions to discover underlying behavioral patterns.
  • Utilize wearable, sophisticated sociometric devices (recording audio, video, and location) for future data collection.
  • Explore applications in collaborative working environments to gain insights into role-playing and group structure.
  • Investigate how informational flow propagates in groups and small communities.

Summary updated

Abstract

Social interactions are a very important component in people’s lives. Social network analysis has become a common technique used to model and quantify the properties of social interactions. In this paper, we propose an integrated framework to explore the characteristics of a social network extracted from multimodal dyadic interactions. For our study, we used a set of videos belonging to New York Times’ Blogging Heads opinion blog. The Social Network is represented as an oriented graph, whose directed links are determined by the Influence Model. The links’ weights are a measure of the “influence” a person has over the other. The states of the Influence Model encode automatically extracted audio/visual features from our videos using state-of-the art algorithms. Our results are reported in terms of accuracy of audio/visual data fusion for speaker segmentation and centrality measures used to characterize the extracted social network.
BibTeX
@inproceedings{escalera2012social,
  author = {Sergio Escalera and Xavier Baró and Jordi Vitrià and Petia Radeva and Bogdan Raducanu},
  title = {{Social Network Extraction and Analysis Based on Multimodal Dyadic Interaction}},
  booktitle = {Sensors},
  year = {2012},
  url = {https://openreview.net/forum?id=OfzurAljk7b},
}
Figure