Abstract
Theory of Mind (ToM)—the ability to attribute mental states such as beliefs, goals, and intentions to others—is a cornerstone of human social intelligence. While Large Vision-Language Models (LVLMs) have shown promise in text-based ToM tasks, these benchmarks often lack the complexity of real-world, embodied interactions. We present SoMi-ToM, a new benchmark designed to evaluate ToM in LVLMs within dynamic, multi-agent Minecraft environments. By requiring agents to collaborate or compete while navigating physical constraints and incomplete information, we reveal a significant performance gap between state-of-the-art models and humans. Our findings highlight critical failures in resource tracking, hierarchical goal inference, and the integration of subjective perspectives.
Introduction
Social intelligence is inherently embodied, multimodal, and multi-perspective. In everyday life, we do not just process text; we observe actions in a physical space, listen to dialogue, and constantly update our beliefs about what others see and know. Traditional ToM evaluations, like the classic "Sally-Anne" test, are often static and fail to capture these dynamics.
To bridge this gap, we introduce the SoMi-ToM benchmark. It places AI agents in a simulated physical environment (Minecraft) where they must achieve complex crafting goals. This setting introduces Theory of Mind challenges where agents must infer the mental states of others based on limited visual evidence and social interactions, such as a teammate secretly attempting to obstruct progress.
The SoMi-ToM Framework
The researchers developed SoMi, an embodied interaction environment built on Minecraft. The architecture utilizes a three-tiered asynchronous communication system (AAct), allowing multiple LVLM-controlled agents to interact autonomously without human intervention.
Agents are endowed with visual perception through periodic first-person screenshots. They must integrate these visual cues with dialogue history and system feedback to make decisions. The evaluation is split into two primary perspectives:
- First-Person Evaluation: The model "plays" an agent and must infer the current state of the world and its teammates (e.g., "What materials does Jane have?") based on its subjective experience.
- Third-Person Evaluation: The model acts as an external observer watching a video of the interaction to infer high-level goals and specific behaviors.
Dataset and Social Dynamics
The SoMi-ToM benchmark comprises 35 unique tasks involving three agents. To test social reasoning, 20 tasks are purely collaborative, while 15 involve "unidirectional obstruction," where one agent secretly delays the team. The dataset includes 363 first-person images, 35 third-person videos (averaging 4.4 minutes), and 1,225 expert-annotated multiple-choice questions. Unlike previous benchmarks like MMToM-QA, SoMi-ToM emphasizes long-horizon video understanding and simultaneous evaluation of multiple perspectives.
Key Insights: Why Models Fail
Our analysis identified several recurring failure modes in current LVLMs:
- The "Intent vs. Reality" Trap: Models frequently prioritize an agent’s stated plan over the actual observed outcome. If an agent says they will collect three logs but only retrieves two, models often incorrectly update their belief to three.
- Hierarchical Blindness: Models struggle to distinguish between sub-goals and final goals. For instance, observing an agent collecting wood might lead a model to conclude the goal is "collect logs," missing the broader objective of "crafting a boat."
- Numerical Reasoning as Social Reasoning: Effective ToM in this context requires precise resource tracking. Models often fail to account for resource consumption during crafting (e.g., 4 planks for a crafting table), leading to erroneous inferences about what others possess.
- Chain-of-Thought (CoT) Variability: While CoT reasoning significantly improved performance for top-tier models like GPT-4o, it proved detrimental for smaller open-source models, which often hallucinated during the reasoning process.
Experimental Results
The performance gap between humans and AI remains substantial across all categories. Humans achieved a baseline accuracy of 90.0% in first-person evaluations and 93.3% in third-person evaluations.
In the first-person state inference task, the best-performing model (GPT-4o with CoT) reached only 59.5% accuracy, leaving a 40.1% gap compared to humans. Models performed slightly better in the third-person goal/behavior tasks, with an average accuracy of 66.9%, but still lagged behind the human baseline by 26.4%. Among the tested models, GPT-4o, Gemini 2.0 Flash, and InternVL2.5 78B were the strongest performers, while smaller models like LLaVA 1.6 13B struggled with basic instruction following.
Conclusion and Future Work
SoMi-ToM highlights the limitations of current LVLMs in complex, embodied social reasoning. While models can follow simple instructions, they lack the robust Theory of Mind required to navigate dynamic environments with incomplete information. Future research will focus on scaling to more complex, long-horizon tasks—such as large-scale construction—and introducing more nuanced social variables, including varying agent personalities and multi-party social backgrounds. Improving numerical reasoning in dynamic contexts will also be a key priority for developing more socially intelligent AI agents.
Summary updated