Abstract
Social reasoning abilities are crucial for AI systems to effectively interpret and respond to multimodal human communication and interaction within social contexts. We introduce Social Genome, the first benchmark for fine-grained, grounded social reasoning abilities of multimodal models. Social Genome contains 272 videos of interactions and 1,486 human-annotated reasoning traces related to inferences about these interactions. These traces contain 5,777 reasoning steps that reference evidence from visual cues, verbal cues, vocal cues, and external knowledge (contextual knowledge external to videos). Social Genome is also the first modeling challenge to study external knowledge in social reasoning. Social Genome computes metrics to holistically evaluate semantic and structural qualities of model-generated social reasoning traces. We demonstrate the utility of Social Genome through experiments with state-of-the-art models, identifying performance gaps and opportunities for future research to improve the grounded social reasoning abilities of multimodal models.
Introduction
Social intelligence remains one of the "final frontiers" for artificial intelligence. While modern multimodal models excel at technical tasks like coding or mathematical problem-solving, interpreting the nuances of human social interaction—such as identifying why a person feels "awkward" or "surprised"—remains notoriously difficult.
Existing benchmarks often rely on Multiple Choice Question Answering (MCQA), which allows models to potentially exploit statistical biases without truly understanding social dynamics. To address this, the authors of SOCIAL GENOME argue that true social reasoning requires grounding: the ability to identify specific multimodal evidence (e.g., a fleeting glance or a sarcastic tone) and integrate it with external knowledge of social norms and cultural expectations.
The Social Genome Benchmark
The SOCIAL GENOME benchmark is designed to evaluate whether models can provide "grounded" reasoning traces alongside their answers. Built upon 272 videos from the Social-IQ 2.0 dataset, it features 1,486 human-annotated reasoning traces comprising 5,777 individual steps. These traces are exceptionally detailed, referencing over 11,000 entities and 5,000 multimodal cues.
Unlike previous datasets, Social Genome requires models to explicitly label the modality of their evidence (visual, verbal, or vocal) and identify when external knowledge is necessary for an inference. This fine-grained approach allows for a systematic evaluation of whether models are actually performing step-by-step reasoning or just guessing.
Evaluation Framework
To assess the quality of model-generated reasoning traces, the researchers introduced several novel metrics:
- Semantic Similarity: Measures how well the model's explanation matches the meaning of human annotations using embeddings.
- DifferenceSequence (DS): A structural metric based on Levenshtein edit distance that checks if the model references modalities in the same order and frequency as humans.
- EmotionMetric: Evaluates the alignment of referenced emotions with human observations.
- External Knowledge Tagging: Specifically checks the model's ability to identify when information outside the video (e.g., social norms) is required.
Key Results and Insights
Experiments with state-of-the-art models like GPT-4o and Gemini 1.5 Flash reveal a significant performance gap:
- The Reasoning Gap: While humans achieve 85.3% accuracy, the best model (Gemini-1.5-Flash) reached 74.4%. However, even the best models achieved less than 50% semantic similarity to human reasoning traces, suggesting they often reach the right answer through flawed reasoning.
- The External Knowledge Challenge: Humans rely on external social norms for approximately 51% of their reasoning steps. Models struggle significantly here, often failing to identify or reference this "common sense" context.
- Modality Struggles: Models are reasonably proficient with visual and verbal cues but falter with vocal cues (prosody and tone) and hierarchical reasoning. Human reasoning is often "forked" and layered, while model traces tend to be flatter and more linear.
- The ICL Paradox: Interestingly, increasing "few-shot" examples often decreased social inference accuracy. This suggests that the informal, non-rigid logic of social reasoning does not benefit from standard prompting techniques as much as mathematical or logical tasks.
Conclusion and Future Work
Social Genome highlights that current multimodal models still lack the depth of grounded social reasoning found in humans. Future research should focus on architectures that can better handle hierarchical social evidence and more effectively integrate vast knowledge banks of social norms. Additionally, expanding the benchmark to include multicultural contexts and longer video durations will be essential for developing truly socially intelligent AI.
Summary updated