Face-to-Face Co-Located Human-Human Social Interaction Analysis using Nonverbal Cues: A Survey

Cigdem Beyan, Alessandro Vinciarelli, Alessio Del BuearXiv 2022 · arXiv · OpenReview

TL;DR

This survey systematically reviews recent research (since 2010) on automatically analyzing nonverbal cues in face-to-face human interactions to understand social phenomena like personality, roles, and group dynamics. It highlights advances in AI, identifies limitations in current datasets and methodologies, and proposes future research directions for more robust and privacy-preserving social interaction analysis.

What Problem Does This Paper Address?

Understanding human behavior, particularly in social contexts, is crucial for applications like social robotics, mental health detection, and human-computer interaction. While Human Activity Recognition (HAR) has focused on recognizing predefined actions, it often misses the "social intelligence" needed to interpret why actions occur and their social meaning. This paper addresses this gap by surveying research that uses nonverbal cues (like speech, gestures, gaze, and facial expressions) to automatically detect and understand complex social and psychological phenomena in co-located human interactions. The challenge lies in effectively capturing and interpreting these subtle cues across diverse settings and using computational methods to infer nuanced social states.

Methods

The paper reviews a wide range of computational methodologies, primarily focusing on machine learning and deep learning approaches.

  • Machine Learning: Support Vector Machines (SVMs) are the most frequently used classifiers, often combined with radial basis function (RBF) or linear kernels. Other common methods include Naive Bayes, k-Nearest Neighbors, Random Forests, and Bayesian Networks.
  • Deep Learning: More recent work leverages deep architectures like Convolutional Neural Networks (CNNs), Fully Connected Neural Networks (FCNNs), Graph Convolutional Networks (GCNNs), and Recurrent Neural Networks (RNNs), including Long Short-Term Memory (LSTM) networks. These are often used for feature extraction or end-to-end detection.
  • Unsupervised Methods: Some studies employ unsupervised approaches like Gaussian Mixture Models (GMMs), Latent Dirichlet Allocation (LDA), and Granger Causality for tasks like engagement analysis or dominance modeling.
  • Multimodal Fusion: A significant trend is the fusion of multiple nonverbal cues (e.g., audio and visual) at both the feature extraction and classifier levels to improve performance.

Datasets

The paper discusses numerous datasets used for training and evaluation, noting that a comprehensive summary of these is a key contribution.

  • Variety of Scenarios: Datasets cover diverse scenarios including meetings (often with 3-4 participants), free-standing conversations, speed-dates, role-playing games, interviews, and even real-world settings like coffee breaks and public spaces.
  • Annotation Types: Annotations are derived from self-assessments (S), external observers (E), or automatic methods (A).
  • Public Availability: The paper notes that while many datasets exist, not all are publicly available, which hinders reproducibility and comparative studies. Table 4 provides a detailed summary of datasets, including their scenarios, group sizes, participant numbers, annotations, sensors, and public availability.

Key Insights and Ideas

  • Nonverbal Cues as Social Traces: The core idea is that nonverbal behaviors are physical manifestations of underlying social and psychological phenomena, making them prime targets for automated analysis.
  • Multimodality is Key: Combining different nonverbal cues (e.g., speech, gaze, body language) significantly improves the accuracy and robustness of detecting social phenomena compared to unimodal approaches.
  • Deep Learning's Rise: Deep learning models, particularly CNNs and LSTMs, have shown superior performance in many tasks, but they haven't been applied to all social phenomena yet.
  • Meetings Dominate: Small group meetings (3-4 people) are the most common setting for research, likely due to easier data collection and annotation.
  • Personality Traits are Popular: Personality trait detection has received the most attention among all social phenomena.
  • Speaking Activity is Ubiquitous: Vocal behavior, especially speaking activity (speaker diarization), is the most frequently used nonverbal cue.

Key Results

  • Improved Performance with Multimodal Features: Numerous studies demonstrate that multimodal features lead to better inference performance than unimodal ones.
  • Deep Learning SOTA: Deep learning architectures have achieved state-of-the-art (SOTA) performance in several areas, though direct comparisons are difficult due to varied datasets.
  • Speaking Activity Dominance: Speaking activity cues are consistently found to be highly effective for detecting dominance and other social traits.
  • Specific Cue Effectiveness: For personality traits, vocal cues (pitch, energy) and VFOA were effective. For hirability, speaking and body activities, along with facial expressions, were important.
  • No End-to-End Pipelines Yet: Despite advances, fully end-to-end systems for detecting social phenomena are still rare.

Limitations and Future Work

  • Dataset Limitations:
    • Many datasets are not publicly available, hindering reproducibility and comparative studies.
  • Existing datasets are often limited in size and may lack annotation reliability analysis.
  • There's a need for large-scale, real-world "in-the-wild" datasets that capture diverse interactions and longer durations.
  • Collecting data across diverse cultural backgrounds is also suggested.
  • Methodological Gaps:
    • Deep learning models have not been applied to all social phenomena (e.g., hirability, group satisfaction).
  • There's a lack of cross-dataset analysis to assess model generalizability.
  • Explainability and interpretability of AI models remain a challenge.
  • Privacy-preserving methods for nonverbal signal processing are largely unexplored.
  • Future Directions:
    • Artificial Intelligence: Explore unsupervised pre-training, domain adaptation, and integrate deep learning for under-researched phenomena. Develop explainability tools (e.g., SHAP, Grad-CAM).
  • Dataset Collection: Create larger, more diverse, and publicly available datasets, focusing on real-world scenarios and long-term interactions. Implement robust annotation reliability analysis.
  • Privacy-Preserving Analysis: Develop methods for privacy-preserving nonverbal signal representations, sensors, and computational models, while maintaining performance.
  • Adaptive Models: Implement online learning and adaptive models to handle real-life deployment challenges like domain shifts and missing sensor data.

Summary updated

Abstract

This work presents a systematic review of recent efforts (since 2010) aimed at automatic analysis of nonverbal cues displayed in face-to-face co-located human-human social interactions. The main reason for focusing on nonverbal cues is that these are the physical, machine detectable traces of social and psychological phenomena. Therefore, detecting and understanding nonverbal cues means, at least to a certain extent, to detect and understand social and psychological phenomena. The covered topics are categorized into three as: a) modeling social traits, such as leadership, dominance, personality traits, b) social role recognition and social relations detection and c) interaction dynamics analysis in terms of group cohesion, empathy, rapport and so forth. We target the co-located interactions, in which the interactants are always humans. The survey covers a wide spectrum of settings and scenarios, including free-standing interactions, meetings, indoor and outdoor social exchanges, dyadic conversations, and crowd dynamics. For each of them, the survey considers the three main elements of nonverbal cues analysis, namely data, sensing approaches and computational methodologies. The goal is to highlight the main advances of the last decade, to point out existing limitations, and to outline future directions.
BibTeX
@misc{beyan2022face,
  author = {Cigdem Beyan and Alessandro Vinciarelli and Alessio Del Bue},
  title = {{Face-to-Face Co-Located Human-Human Social Interaction Analysis using Nonverbal Cues: A Survey}},
  year = {2022},
  eprint = {2207.10574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.10574},
}
Figure