TL;DR
Standard Theory-of-Mind (ToM) benchmarks for vision-language models (VLMs) often assume that asking a model to report another person's perspective is a neutral act. This paper demonstrates that the act of asking for a report can paradoxically trigger egocentric errors—causing models to act on their own view instead of the other person's—even when the models correctly report what the other person sees.
What Problem Does This Paper Address?
To work effectively alongside humans, VLMs must possess "Theory of Mind": the ability to understand that others may have different perspectives, knowledge, or intentions. A classic test for this is the Director Task, where a "director" asks a "participant" (the model) to move an object. Crucially, some objects are hidden from the director's view by a partition. If the director asks for "the largest block" and the absolute largest block is hidden from them, the correct response is to move the largest block visible to the director.
Current evaluation protocols typically ask the model two questions in one prompt:
- "Which blocks can the director see?" (The Report)
- "Which block will you move?" (The Action)
Researchers have often interpreted a correct report followed by an incorrect (egocentric) action as a fundamental failure of the model to use the perspective it just worked out. This paper challenges that interpretation, suggesting the failure might be an artifact of the test design itself.
Methods
The authors created a rendered 3D version of the Director Task featuring a tabletop with numbered, colored blocks of varying sizes. They tested ten different models, including various versions of GPT-4o (referred to as GPT-5.6 and GPT-6 in the paper's internal naming), Grok-4.6, and open-weight models like Qwen3.8-27B.
Their core methodology relies on paired prompts:
- Base Prompt: Asks both the visibility question and the move question.
- Ablated Prompt: Asks only the move question.
- Constraint Prompt: Explicitly states that the director only knows about and refers to visible blocks.
By comparing these, they could isolate the effect of the visibility question itself. For open-weight models, they also analyzed reasoning traces (Chain-of-Thought) to see why the models made certain choices.
Datasets
The study uses two primary synthetic datasets:
- Main Set: 160 scenes where the largest block from the model's view is always hidden from the director.
- Rotated Set: 160 scenes where the hidden block varies (largest, middle, or smallest) to check for simple biases.
- Rearranged Digits Set: A control set where block numbers were shuffled to ensure models weren't just picking the block with the largest digit.
Key Insights and Ideas
- The "Question-Behavior" Effect: Just like in human surveys, the sequence of questions matters. Asking about the director's view first seems to "use up" that topic. When the model is then asked what it will move, it may interpret this as a request to act on its own, unrestricted view.
- Correct Report, Wrong Action $\neq$ ToM Failure: Models often correctly identified the hidden blocks in the first part of the prompt but then ignored that information when choosing which block to move. Reasoning traces showed models explicitly stating "the director cannot see block X" but then concluding "I will move block X because it is the largest."
- Pragmatic Shift: Rewording the request from "Which number will you move?" to "Which number does the director want you to move?" significantly reduced errors. This suggests the models aren't "blind" to perspective; they are just sensitive to how the task is framed.
- The "Combined Constraint" Fix: When the prompt explicitly states the rules of the game (the director doesn't know about hidden blocks and their words only refer to visible ones), egocentric errors virtually disappear (dropping to <1%).
Key Results
- Massive Error Spikes: Adding the visibility question increased the rate of choosing the hidden (incorrect) block by 12 to 88 percentage points across six GPT models.
- Consistency in Accuracy: Visibility reports were generally accurate (often >90%) regardless of whether the subsequent action was right or wrong.
- Baseline Matters: Without the visibility question, most GPT models chose the hidden block in less than 7% of cases. The "standard" evaluation protocol was actually making the models look much worse at ToM than they actually are.
Limitations and Future Work
- Single Paradigm: The study focused exclusively on the Director Task with block labels. It remains to be seen if these findings generalize to other ToM tasks like False Belief tests.
- English Only: The prompts were exclusively in English.
- Reasoning Effort: Models were tested at "medium" reasoning effort; higher-compute reasoning might show different dynamics.
- Human Comparison: While the paper notes that humans also make egocentric errors in the Director Task, it does not test whether asking humans the visibility question first would similarly change their error rates.
The authors conclude that any future ToM benchmark must include a baseline that asks for the action without first eliciting a perspective report to ensure the test isn't measuring its own interference.
Summary updated