Abstract
Social reasoning abilities are crucial for AI systems to effectively interpret and respond to multimodal human communication and interaction within social contexts. We introduce SOCIAL GENOME, the first benchmark for fine-grained, grounded social reasoning abilities of multimodal models. SOCIAL GENOME contains 272 videos of interactions and 1,486 human-annotated reasoning traces related to inferences about these interactions. These traces contain 5,777 reasoning steps that reference evidence from visual cues, verbal cues, vocal cues, and external knowledge (contextual knowledge external to videos). SOCIAL GENOME is also the first modeling challenge to study external knowledge in social reasoning. SOCIAL GENOME computes metrics to holistically evaluate semantic and structural qualities of model-generated social reasoning traces. We demonstrate the utility of SOCIAL GENOME through experiments with state-of-the-art models, identifying performance gaps and opportunities for future research to improve the grounded social reasoning abilities of multimodal models.
Introduction
Social intelligence remains a "holy grail" for artificial intelligence, particularly for applications requiring nuanced human-AI interaction such as healthcare assistants or social robots. While existing benchmarks evaluate whether a model can correctly answer a social question (e.g., "Is this person being sarcastic?"), they often fail to measure the underlying logic. This creates a risk where models may provide correct answers based on statistical shortcuts rather than genuine social understanding.
To address this, we present SOCIAL GENOME, a benchmark designed to evaluate how well multimodal models ground their social inferences in specific evidence. By requiring models to produce "reasoning traces," we can move beyond simple accuracy and analyze the fine-grained cues—visual, vocal, and verbal—that drive social interpretation.
The SOCIAL GENOME Benchmark
Built upon 272 videos from the Social-IQ 2.0 dataset, SOCIAL GENOME provides a dense repository of human-annotated social logic. The dataset includes:
- 1,486 Reasoning Traces: Detailed step-by-step explanations of social inferences.
- 5,777 Reasoning Steps: Each step is tagged by modality (visual, verbal, vocal) and whether it requires external knowledge.
- High Entity Density: The traces reference over 11,000 entities (people, objects, concepts) and 2,900 specific external knowledge observations.
This granularity allows for a rigorous examination of how models bridge the gap between raw sensory input and high-level social conclusions.
Methodology: Measuring the "Why"
In the SOCIAL GENOME task, a model is presented with a video and a social question. It must provide both a final answer and a structured reasoning trace. To evaluate these traces, we developed 12 metrics across two primary categories:
- Semantic Metrics: These utilize embeddings (specifically all-MiniLM-L6-v2) to measure the conceptual alignment between model-generated traces and human-annotated logic at both the global and step-by-step levels.
- Structural Metrics: Using an adapted Levenshtein distance called DifferenceSequence, we evaluate whether models reference different modalities in the same logical order and frequency as humans.
We benchmarked seven state-of-the-art models, including GPT-4o, Gemini 1.5 Flash, and open-source models like LLaVA-Video and LongVA, using zero-shot and few-shot in-context learning (ICL) settings.
Key Insights and Findings
The Reasoning Gap
While top-tier models like Gemini 1.5 Flash (74.4%) and GPT-4o (71.0%) perform respectably in terms of answer accuracy, they lag significantly behind the human baseline of 85.3%. More importantly, even when models arrive at the correct answer, their reasoning traces show only about 50% semantic similarity to human logic. This suggests that models are often "guessing right" for the wrong reasons.
The In-Context Learning (ICL) Paradox
In many AI domains, providing more examples (few-shot learning) improves performance. However, in SOCIAL GENOME, we observed that increasing the number of examples often decreased model accuracy. Social rules are frequently implicit and context-dependent, making them harder to distill from a few samples compared to the formal logic found in mathematics or coding.
The External Knowledge and Vocal Cue Challenges
Humans rely on external knowledge—unwritten social norms and cultural context—in 51% of their reasoning steps. Models struggled to integrate this context, failing to connect a physical action (like a high-five) to its social meaning (celebration or agreement). Furthermore, while models are improving at visual and verbal tasks, their ability to incorporate vocal cues (prosody and tone) remains a significant weakness.
Hierarchical vs. Linear Logic
Human social reasoning is often hierarchical; we combine multiple low-level cues (a squint, a specific tone) into intermediate conclusions (suspicion) before reaching a final inference. In contrast, current models tend to produce "flat," linear traces that overlook these essential intermediate steps, often leading to "error propagation" where a single misidentified behavior invalidates the entire logical chain.
Conclusion and Future Work
SOCIAL GENOME reveals that while multimodal models are becoming more capable, their grounded social reasoning remains limited. Future research must address the "reasoning gap" by improving how models integrate external knowledge and vocal prosody.
Limitations of the current study include a focus on English-language interactions and Western-centric social norms. We advocate for the community-driven curation of multicultural interaction data to ensure that the next generation of socially intelligent AI is globally inclusive and truly grounded in the complexities of human behavior.
Summary updated