TL;DR
This paper analyzes the social intelligence of Large Language Models (LLMs) on video-based question-answering tasks, revealing that while LLMs like GPT-40 achieve human-comparable accuracy, they differ significantly in their reasoning and reliance on modalities. The study highlights limitations in current social intelligence benchmarks and suggests future directions for more robust evaluation.
What Problem Does This Paper Address?
The increasing integration of AI into human interactions necessitates AI systems with social intelligence – the ability to understand human behavior, emotions, and social dynamics. While datasets like Social-IQ have been developed to evaluate this, they often suffer from simplistic question-answer formats, lack justifications for answers, and rely on end-to-end models that limit interpretability. This paper addresses the need for a deeper understanding of how LLMs perform on these complex social intelligence tasks, comparing their reasoning and response patterns to humans and identifying shortcomings in existing evaluation methods.
Methods
The study employs a zero-shot approach, evaluating LLMs (specifically GPT-4 Turbo and GPT-40) on the Social-IQ 2.0 dataset. This involves providing the models with video content (frames and/or captions) and questions, and analyzing their chosen answers and provided justifications. The researchers also created an auxiliary dataset (Social-IQ 2.0 Sub) with human responses and their reasoning to enable a more detailed comparison. An ablation study was conducted to investigate the impact of various input factors, including data types, frame count, sampling strategies, and caption granularity, on LLM performance.
Datasets
- Social-IQ 2.0: A video-based question-answering benchmark designed to assess social intelligence. It features over 1000 videos with multiple-choice questions about human interactions, intentions, and emotions.
- Social-IQ 2.0 Sub: An auxiliary dataset created by the authors, consisting of 200 videos from Social-IQ 2.0, annotated with human responses and their reasoning. This dataset was crucial for detailed qualitative analysis.
Key Insights and Ideas
- LLMs are catching up, but reasoning differs: LLMs like GPT-40 are achieving accuracy comparable to humans on social intelligence tasks, but their underlying reasoning processes are distinct. Humans leverage a broader range of multimodal cues and contextual knowledge, while LLMs often rely more heavily on dialogue transcripts.
- "Unanswerable" is a key differentiator: GPT-4 Turbo, in particular, tends to mark more questions as "unanswerable" when information is insufficient, a more cautious approach than humans who might attempt an answer with more inference. GPT-40 shows a better balance, closer to human tendencies.
- Current benchmarks have limitations: The Social-IQ 2.0 dataset's reliance on multiple-choice answers without ground truth for reasoning hinders a full evaluation of LLM capabilities. The ambiguity and subjectivity inherent in social intelligence tasks also pose challenges for creating definitive correct answers.
- Multimodality is crucial, but not always straightforward: While GPT-40's multimodal capabilities improve its performance, simply increasing the number of frames or using detailed captions doesn't always lead to better results, suggesting that the relevance of information is more critical than its quantity.
- Option discriminability matters: The clarity and distinguishability of answer options significantly impact accuracy, highlighting the need for well-designed QA pairs in benchmarks.
Key Results
- Zero-shot LLM performance: GPT-4 Turbo and GPT-40 achieved accuracy comparable to state-of-the-art trained methods on the Social-IQ 2.0 benchmark, demonstrating strong zero-shot capabilities.
- Accuracy comparison: GPT-40 achieved higher overall accuracy (77.79%) than GPT-4 Turbo (63.61%) when using a combination of captions and images. GPT-40's accuracy was also closer to human accuracy (76.87%) in certain configurations.
- Response patterns: Humans tend to utilize non-verbal cues and broader context, while GPT-4 Turbo heavily relies on dialogue and can struggle with visual information. GPT-40 shows a better integration of multimodal information.
- Question type analysis: Humans and GPT-40 showed fewer "unanswerable" responses for timestamp-based questions. GPT-40 demonstrated higher accuracy on "Cause" questions, potentially due to its literal interpretation and reduced susceptibility to subjective biases.
- Ablation study findings: Using captions alone resulted in significantly lower accuracy for humans compared to LLMs. For GPT-4 Turbo, adding visual information sometimes introduced noise. For GPT-40, increasing frames beyond a certain point didn't consistently improve accuracy, and image-based input was more effective than captions for higher frame counts.
Limitations and Future Work
- Limitations:
- The Social-IQ 2.0 dataset's "unanswerable" criteria are subjective.
- The dataset lacks sufficient evaluation metrics for a comprehensive assessment of social intelligence.
- LLMs still struggle with integrating multimodal information effectively and critiquing question design.
- The reliance on dialogue information for LLMs can lead to errors when visual or auditory cues are critical.
- Future Work:
- Develop new methodologies and, more importantly, new benchmarks with denser annotations and more robust evaluation metrics.
- Create QA tasks that include rationales to enable systems to handle ambiguity and develop human-like reasoning.
- Incorporate evaluation metrics for social appropriateness, scene context understanding, and reasoning consistency.
- Explore modular approaches for LLMs to generate intermediate outputs for detailed analysis of complex, long-duration videos.
- Realize applications such as natural communication with caregivers, improved customer service, and emotional coaching for specific populations.
Summary updated