SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions

Xianzhe Fan, Xuhui Zhou, Chuanyang Jin, Kolby Nottingham, Hao Zhu, Maarten Sap

Abstract

Theory of Mind (ToM)—the ability to attribute mental states such as beliefs, goals, and intentions to others—is a cornerstone of human social intelligence. While Large Vision-Language Models (LVLMs) have shown promise in text-based ToM tasks, these benchmarks often lack the complexity of real-world, embodied interactions. We present SoMi-ToM, a new benchmark designed to evaluate ToM in LVLMs within dynamic, multi-agent Minecraft environments. By requiring agents to collaborate or compete while navigating physical constraints and incomplete information, we reveal a significant performance gap between state-of-the-art models and humans. Our findings highlight critical failures in resource tracking, hierarchical goal inference, and the integration of subjective perspectives.

Introduction

Social intelligence is inherently embodied, multimodal, and multi-perspective. In everyday life, we do not just process text; we observe actions in a physical space, listen to dialogue, and constantly update our beliefs about what others see and know. Traditional ToM evaluations, like the classic "Sally-Anne" test, are often static and fail to capture these dynamics.

To bridge this gap, we introduce the SoMi-ToM benchmark. It places AI agents in a simulated physical environment (Minecraft) where they must achieve complex crafting goals. This setting introduces Theory of Mind challenges where agents must infer the mental states of others based on limited visual evidence and social interactions, such as a teammate secretly attempting to obstruct progress.

The SoMi-ToM Framework

The researchers developed SoMi, an embodied interaction environment built on Minecraft. The architecture utilizes a three-tiered asynchronous communication system (AAct), allowing multiple LVLM-controlled agents to interact autonomously without human intervention.

Overview of the SoMi-ToM framework, illustrating the interaction between materials, goals, social dynamics, and the multi-perspective evaluation process.

Agents are endowed with visual perception through periodic first-person screenshots. They must integrate these visual cues with dialogue history and system feedback to make decisions. The evaluation is split into two primary perspectives:

  1. First-Person Evaluation: The model "plays" an agent and must infer the current state of the world and its teammates (e.g., "What materials does Jane have?") based on its subjective experience.
  2. Third-Person Evaluation: The model acts as an external observer watching a video of the interaction to infer high-level goals and specific behaviors.

Dataset and Social Dynamics

The SoMi-ToM benchmark comprises 35 unique tasks involving three agents. To test social reasoning, 20 tasks are purely collaborative, while 15 involve "unidirectional obstruction," where one agent secretly delays the team. The dataset includes 363 first-person images, 35 third-person videos (averaging 4.4 minutes), and 1,225 expert-annotated multiple-choice questions. Unlike previous benchmarks like MMToM-QA, SoMi-ToM emphasizes long-horizon video understanding and simultaneous evaluation of multiple perspectives.

Key Insights: Why Models Fail

Our analysis identified several recurring failure modes in current LVLMs:

  • The "Intent vs. Reality" Trap: Models frequently prioritize an agent’s stated plan over the actual observed outcome. If an agent says they will collect three logs but only retrieves two, models often incorrectly update their belief to three.
  • Hierarchical Blindness: Models struggle to distinguish between sub-goals and final goals. For instance, observing an agent collecting wood might lead a model to conclude the goal is "collect logs," missing the broader objective of "crafting a boat."
  • Numerical Reasoning as Social Reasoning: Effective ToM in this context requires precise resource tracking. Models often fail to account for resource consumption during crafting (e.g., 4 planks for a crafting table), leading to erroneous inferences about what others possess.
  • Chain-of-Thought (CoT) Variability: While CoT reasoning significantly improved performance for top-tier models like GPT-4o, it proved detrimental for smaller open-source models, which often hallucinated during the reasoning process.

Experimental Results

The performance gap between humans and AI remains substantial across all categories. Humans achieved a baseline accuracy of 90.0% in first-person evaluations and 93.3% in third-person evaluations.

Comparison of human performance versus various LVLMs on Self-ToM and Others' ToM reasoning tasks, showing the impact of Chain-of-Thought (CoT) prompting.

In the first-person state inference task, the best-performing model (GPT-4o with CoT) reached only 59.5% accuracy, leaving a 40.1% gap compared to humans. Models performed slightly better in the third-person goal/behavior tasks, with an average accuracy of 66.9%, but still lagged behind the human baseline by 26.4%. Among the tested models, GPT-4o, Gemini 2.0 Flash, and InternVL2.5 78B were the strongest performers, while smaller models like LLaVA 1.6 13B struggled with basic instruction following.

Conclusion and Future Work

SoMi-ToM highlights the limitations of current LVLMs in complex, embodied social reasoning. While models can follow simple instructions, they lack the robust Theory of Mind required to navigate dynamic environments with incomplete information. Future research will focus on scaling to more complex, long-horizon tasks—such as large-scale construction—and introducing more nuanced social variables, including varying agent personalities and multi-party social backgrounds. Improving numerical reasoning in dynamic contexts will also be a key priority for developing more socially intelligent AI agents.

Summary updated

Abstract

Humans continuously infer the states, goals, and behaviors of others by perceiving their surroundings in dynamic, real-world social interactions. However, most Theory of Mind (ToM) benchmarks only evaluate static, text-based scenarios, which have a significant gap compared to real interactions. We propose the SoMi-ToM benchmark, designed to evaluate multi-perspective ToM in embodied multi-agent complex social interactions. This benchmark is based on rich multimodal interaction data generated by the interaction environment SoMi, covering diverse crafting goals and social relationships. Our framework supports multi-level evaluation: (1) first-person evaluation provides multimodal (visual, dialogue, action, etc.) input from a first-person perspective during a task for real-time state inference, (2) third-person evaluation provides complete third-person perspective video and text records after a task for goal and behavior inference. This evaluation method allows for a more comprehensive examination of a model's ToM capabilities from both the subjective immediate experience and the objective global observation. We constructed a challenging dataset containing 35 third-person perspective videos, 363 first-person perspective images, and 1225 expert-annotated multiple-choice questions (three options). On this dataset, we systematically evaluated the performance of human subjects and several state-of-the-art large vision-language models (LVLMs). The results show that LVLMs perform significantly worse than humans on SoMi-ToM: the average accuracy gap between humans and models is 40.1% in first-person evaluation and 26.4% in third-person evaluation. This indicates that future LVLMs need to further improve their ToM capabilities in embodied, complex social interactions.
BibTeX
@misc{fan2025somi,
  author = {Xianzhe Fan and Xuhui Zhou and Chuanyang Jin and Kolby Nottingham and Hao Zhu and Maarten Sap},
  title = {{SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions}},
  year = {2025},
  eprint = {2506.23046},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.23046},
}
Figure