Detecting interaction links in a collaborating group using manually annotated data

Shobhit Mathur, Marshall Scott Poole, Feniosky Peña-Mora, Mark Hasegawa-Johnson, Noshir S. ContractorSoc. Networks 2012 · DOI

TL;DR

This paper presents a novel two-stage approach to automatically detect interaction links within collaborating groups by combining manual annotation of non-verbal cues with machine learning. The method leverages visual cues like gaze, posture, and proximity to build a Dynamic Bayesian Network, offering a more scalable way to study dynamic social networks than traditional observation methods.

What Problem Does This Paper Address?

Understanding how people interact and form connections within groups is crucial for fields ranging from organizational psychology to social network analysis. Traditional methods for studying these interactions, like direct observation, are incredibly time-consuming and labor-intensive, especially when trying to capture the dynamic nature of these links over time. This becomes even more challenging with larger groups. The paper aims to bridge this gap by developing an automated system that can infer interaction links from video data, reducing the manual effort required and enabling the study of more complex, dynamic group interactions.

Methods

The paper proposes a two-stage methodology:

  1. Manual Annotation: A web-based interface was developed to allow human coders to manually annotate key non-verbal cues from video snapshots. These cues include:
  2. Location: The spatial position of each group member within a grid.
  3. Head Orientation: The direction of a member's head, used to infer gaze.
  4. Bodily Posture: Whether a member is sitting/standing and looking up/down.
  5. Speaking (Y/N): Whether a member is actively speaking.
  6. Links: The criterion variable, indicating if two members are interacting.

    This manual annotation stage is crucial for generating the ground truth data needed to train the machine learning model.

  7. Machine Learning for Link Inference: The manually annotated data is used to train a Dynamic Bayesian Network (DBN). The DBN models the temporal dependencies between interaction links and observed cues.

  8. Connectors: The system first infers "connectors" from the annotated cues:
  9. Gaze: Modeled using Gaussian mixtures and Bayes' law, considering head orientation and a "view volume." * Conversational Distance: Inferred from the spatial proximity of members, defining "neighbors." * Speaking: Directly used as a cue.
  10. Dynamic Bayesian Network (DBN): The DBN represents links (L) as hidden variables, and gaze (G), speaking (S), and neighborhood (N) as observed variables. It models the stochastic dependency of links on these cues and on previous link states.
  11. Parameter Estimation: The Bayes Net Toolkit (BNT) is used to estimate the DBN's parameters using the Expectation Maximization (EM) algorithm.
  12. Inference: The Viterbi algorithm is adapted to perform inference, finding the most probable sequence of link states over time.

Datasets

The primary dataset used in this study was generated from video recordings of eight emergency responders planning a response to a simulated emergency situation. This data was collected from a real-world setting at the Illinois Fire Service Institute (IFSI). The video was captured using four cameras, and the session lasted approximately 2.5 hours. This dataset was manually annotated by 12 coders using the custom web-based interface. No new benchmark datasets were introduced; rather, a real-world scenario was used for annotation.

Key Insights and Ideas

  • Leveraging Non-Verbal Cues: The core idea is that subtle non-verbal cues (gaze, posture, proximity) are strong indicators of interaction, and these can be automated with computer vision.
  • Two-Stage Approach: The combination of manual annotation for ground truth and machine learning for inference is a robust strategy for tackling complex observational data.
  • Dynamic Networks: The use of a DBN is critical for capturing the fluid, time-varying nature of social interactions, moving beyond static network snapshots.
  • Scalability Potential: By automating the detection of cues, the system has the potential to scale to much larger groups and longer observation periods than manual methods.
  • Importance of "Connectors": Breaking down the problem into inferring gaze, distance, and speaking status before predicting links simplifies the DBN structure and improves interpretability.

Key Results

  • Gaze Detection: The automatic gaze detection algorithm achieved 68% accuracy, comparable to previous work, and importantly, handled non-stationary members in a group setting.
  • Link Detection Accuracy: The link detection algorithm achieved recall without error ranging from 74.4% to 85.1% depending on the input accuracy (manual vs. automatic gaze, annotated vs. video-based speakers). While slightly lower than some controlled lab studies, this is considered adequate given the complexity of real-world settings and group movement.
  • Annotation Efficiency: The manual annotation process, while labor-intensive, was found to be relatively fast (average 73 seconds per snapshot for five attributes and linkage), significantly speeding up data preparation compared to unstructured coding.
  • Reliability of Annotations: Intercoder reliability measures (Fleiss' Kappa, Cronbach's Alpha) indicated substantial agreement for location, bodily posture, and head orientation, though speaking and link detection showed lower reliability, highlighting areas for improvement.

Limitations and Future Work

  • Manual Annotation Bottleneck: The primary limitation acknowledged is the reliance on manual annotation for training data, which is still time-consuming.
  • Accuracy of Cues: The accuracy of the link detection is directly dependent on the accuracy of the inferred cues (gaze, speaking). Lower accuracy in these areas (e.g., speaking detection) impacts overall performance.
  • Simultaneous Speech Handling: The algorithm simplified simultaneous speech by ignoring short utterances or the interrupter's speech until the floor was yielded, which might miss some nuances.
  • Future Work: The authors suggest that advancements in video recognition technology are key to fully automating the cue detection process, leading to a fully automated link detection system. This would enable the study of larger and more complex social dynamics in real-time. They also note that the software for link detection is available from the first author.

Summary updated

BibTeX
@article{mathur2012detecting,
  author = {Shobhit Mathur and Marshall Scott Poole and Feniosky Peña-Mora and Mark Hasegawa-Johnson and Noshir S. Contractor},
  title = {{Detecting interaction links in a collaborating group using manually annotated data}},
  journal = {Soc. Networks},
  year = {2012},
  doi = {10.1016/j.socnet.2012.04.002},
}
Figure