A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans

Erika Mori, Yue Qiu, Hirokatsu Kataoka, Yoshimitsu AokiSensors 2025 · DOI

TL;DR

This paper analyzes the social intelligence of Large Language Models (LLMs) on video-based question-answering tasks, revealing that while LLMs like GPT-40 achieve human-comparable accuracy, they differ significantly in their reasoning and reliance on modalities. The study highlights limitations in current social intelligence benchmarks and suggests future directions for more robust evaluation.

What Problem Does This Paper Address?

The increasing integration of AI into human interactions necessitates AI systems with social intelligence – the ability to understand human behavior, emotions, and social dynamics. While datasets like Social-IQ have been developed to evaluate this, they often suffer from simplistic question-answer formats, lack justifications for answers, and rely on end-to-end models that limit interpretability. This paper addresses the need for a deeper understanding of how LLMs perform on these complex social intelligence tasks, comparing their reasoning and response patterns to humans and identifying shortcomings in existing evaluation methods.

Methods

The study employs a zero-shot approach, evaluating LLMs (specifically GPT-4 Turbo and GPT-40) on the Social-IQ 2.0 dataset. This involves providing the models with video content (frames and/or captions) and questions, and analyzing their chosen answers and provided justifications. The researchers also created an auxiliary dataset (Social-IQ 2.0 Sub) with human responses and their reasoning to enable a more detailed comparison. An ablation study was conducted to investigate the impact of various input factors, including data types, frame count, sampling strategies, and caption granularity, on LLM performance.

Datasets

  • Social-IQ 2.0: A video-based question-answering benchmark designed to assess social intelligence. It features over 1000 videos with multiple-choice questions about human interactions, intentions, and emotions.
  • Social-IQ 2.0 Sub: An auxiliary dataset created by the authors, consisting of 200 videos from Social-IQ 2.0, annotated with human responses and their reasoning. This dataset was crucial for detailed qualitative analysis.

Key Insights and Ideas

  • LLMs are catching up, but reasoning differs: LLMs like GPT-40 are achieving accuracy comparable to humans on social intelligence tasks, but their underlying reasoning processes are distinct. Humans leverage a broader range of multimodal cues and contextual knowledge, while LLMs often rely more heavily on dialogue transcripts.
  • "Unanswerable" is a key differentiator: GPT-4 Turbo, in particular, tends to mark more questions as "unanswerable" when information is insufficient, a more cautious approach than humans who might attempt an answer with more inference. GPT-40 shows a better balance, closer to human tendencies.
  • Current benchmarks have limitations: The Social-IQ 2.0 dataset's reliance on multiple-choice answers without ground truth for reasoning hinders a full evaluation of LLM capabilities. The ambiguity and subjectivity inherent in social intelligence tasks also pose challenges for creating definitive correct answers.
  • Multimodality is crucial, but not always straightforward: While GPT-40's multimodal capabilities improve its performance, simply increasing the number of frames or using detailed captions doesn't always lead to better results, suggesting that the relevance of information is more critical than its quantity.
  • Option discriminability matters: The clarity and distinguishability of answer options significantly impact accuracy, highlighting the need for well-designed QA pairs in benchmarks.

Key Results

  • Zero-shot LLM performance: GPT-4 Turbo and GPT-40 achieved accuracy comparable to state-of-the-art trained methods on the Social-IQ 2.0 benchmark, demonstrating strong zero-shot capabilities.
  • Accuracy comparison: GPT-40 achieved higher overall accuracy (77.79%) than GPT-4 Turbo (63.61%) when using a combination of captions and images. GPT-40's accuracy was also closer to human accuracy (76.87%) in certain configurations.
  • Response patterns: Humans tend to utilize non-verbal cues and broader context, while GPT-4 Turbo heavily relies on dialogue and can struggle with visual information. GPT-40 shows a better integration of multimodal information.
  • Question type analysis: Humans and GPT-40 showed fewer "unanswerable" responses for timestamp-based questions. GPT-40 demonstrated higher accuracy on "Cause" questions, potentially due to its literal interpretation and reduced susceptibility to subjective biases.
  • Ablation study findings: Using captions alone resulted in significantly lower accuracy for humans compared to LLMs. For GPT-4 Turbo, adding visual information sometimes introduced noise. For GPT-40, increasing frames beyond a certain point didn't consistently improve accuracy, and image-based input was more effective than captions for higher frame counts.

Limitations and Future Work

  • Limitations:
    • The Social-IQ 2.0 dataset's "unanswerable" criteria are subjective.
  • The dataset lacks sufficient evaluation metrics for a comprehensive assessment of social intelligence.
  • LLMs still struggle with integrating multimodal information effectively and critiquing question design.
  • The reliance on dialogue information for LLMs can lead to errors when visual or auditory cues are critical.
  • Future Work:
    • Develop new methodologies and, more importantly, new benchmarks with denser annotations and more robust evaluation metrics.
  • Create QA tasks that include rationales to enable systems to handle ambiguity and develop human-like reasoning.
  • Incorporate evaluation metrics for social appropriateness, scene context understanding, and reasoning consistency.
  • Explore modular approaches for LLMs to generate intermediate outputs for detailed analysis of complex, long-duration videos.
  • Realize applications such as natural communication with caregivers, improved customer service, and emotional coaching for specific populations.

Summary updated

Abstract

In recent years, advancements in the interaction and collaboration between humans and have garnered significant attention. Social intelligence plays a crucial role in facilitating natural interactions and seamless communication between humans and Artificial Intelligence (AI). To assess AI's ability to understand human interactions and the components necessary for such comprehension, datasets like Social-IQ have been developed. However, these datasets often rely on a simplistic question-and-answer format and lack justifications for the provided answers. Furthermore, existing methods typically produce direct answers by selecting from predefined choices without generating intermediate outputs, which hampers interpretability and reliability. To address these limitations, we conducted a comprehensive evaluation of AI methods on a video-based Question Answering (QA) benchmark focused on human interactions, leveraging additional annotations related to human responses. Our analysis highlights significant differences between human and AI response patterns and underscores critical shortcomings in current benchmarks. We anticipate that these findings will guide the creation of more advanced datasets and represent an important step toward achieving natural communication between humans and AI.
BibTeX
@inproceedings{mori2025comprehensive,
  author = {Erika Mori and Yue Qiu and Hirokatsu Kataoka and Yoshimitsu Aoki},
  title = {{A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25020477},
}
Figure