Asking What a Director Sees Can Change What Vision-Language Models Choose

Jiaxuan Guo, Xingyu Ren, Kejia Zhang, Jingxin Yang, Youran Sun, Haizhao YangarXiv 2026 · arXiv

TL;DR

Standard Theory-of-Mind (ToM) benchmarks for vision-language models (VLMs) often assume that asking a model to report another person's perspective is a neutral act. This paper demonstrates that the act of asking for a report can paradoxically trigger egocentric errors—causing models to act on their own view instead of the other person's—even when the models correctly report what the other person sees.

What Problem Does This Paper Address?

To work effectively alongside humans, VLMs must possess "Theory of Mind": the ability to understand that others may have different perspectives, knowledge, or intentions. A classic test for this is the Director Task, where a "director" asks a "participant" (the model) to move an object. Crucially, some objects are hidden from the director's view by a partition. If the director asks for "the largest block" and the absolute largest block is hidden from them, the correct response is to move the largest block visible to the director.

Current evaluation protocols typically ask the model two questions in one prompt:

  1. "Which blocks can the director see?" (The Report)
  2. "Which block will you move?" (The Action)

Researchers have often interpreted a correct report followed by an incorrect (egocentric) action as a fundamental failure of the model to use the perspective it just worked out. This paper challenges that interpretation, suggesting the failure might be an artifact of the test design itself.

Figure 1: One scene, the first in scene order in which the largest block the director can see also looks the largest of them from where the director stands. (A nearer block can look larger than a bigger one farther away; Appendix B.) (a) The partition hides block 2, the largest block from the model’s viewpoint. Of the blocks the director sees, 1 and 8, block 1 is the largest: the director-view block and the keyed answer. (b) The three MOVE prompts compared, abbreviated from Table 1. (c) gpt-5.6-terra’s block choice in each of its five replies to each prompt, the hidden block in bold. Without the question it chooses the hidden block in two of five replies on this scene, more often than over all scenes (Figure 3). The first two rows come from one batch and the third from another (§3).

Methods

The authors created a rendered 3D version of the Director Task featuring a tabletop with numbered, colored blocks of varying sizes. They tested ten different models, including various versions of GPT-4o (referred to as GPT-5.6 and GPT-6 in the paper's internal naming), Grok-4.6, and open-weight models like Qwen3.8-27B.

Their core methodology relies on paired prompts:

  • Base Prompt: Asks both the visibility question and the move question.
  • Ablated Prompt: Asks only the move question.
  • Constraint Prompt: Explicitly states that the director only knows about and refers to visible blocks.

By comparing these, they could isolate the effect of the visibility question itself. For open-weight models, they also analyzed reasoning traces (Chain-of-Thought) to see why the models made certain choices.

Datasets

The study uses two primary synthetic datasets:

  • Main Set: 160 scenes where the largest block from the model's view is always hidden from the director.
  • Rotated Set: 160 scenes where the hidden block varies (largest, middle, or smallest) to check for simple biases.
  • Rearranged Digits Set: A control set where block numbers were shuffled to ensure models weren't just picking the block with the largest digit.

Key Insights and Ideas

  • The "Question-Behavior" Effect: Just like in human surveys, the sequence of questions matters. Asking about the director's view first seems to "use up" that topic. When the model is then asked what it will move, it may interpret this as a request to act on its own, unrestricted view.
  • Correct Report, Wrong Action $\neq$ ToM Failure: Models often correctly identified the hidden blocks in the first part of the prompt but then ignored that information when choosing which block to move. Reasoning traces showed models explicitly stating "the director cannot see block X" but then concluding "I will move block X because it is the largest."
  • Pragmatic Shift: Rewording the request from "Which number will you move?" to "Which number does the director want you to move?" significantly reduced errors. This suggests the models aren't "blind" to perspective; they are just sensitive to how the task is framed.
  • The "Combined Constraint" Fix: When the prompt explicitly states the rules of the game (the director doesn't know about hidden blocks and their words only refer to visible ones), egocentric errors virtually disappear (dropping to <1%).
Figure 2: The comparisons of §§4–6, one per line, and what the hypothesis predicts. Each mark is one model’s estimate, in points or, in the right-hand panel, in percent: filled where it fits the prediction, hollow where it does neither and a cross where it goes against it, by the rule of §3 or, for the knowledge question, as §5 explains. Grey shading marks the predicted range, and a short vertical line, in a model’s colour, the estimate that the model’s mark should be less than. The GPT models are black, Qwen3.8-27B vermillion, Muse Glimmer-30B blue and grok-4.6 orange. The right column names the tables that hold the estimates; variants: Table 1.

Key Results

  • Massive Error Spikes: Adding the visibility question increased the rate of choosing the hidden (incorrect) block by 12 to 88 percentage points across six GPT models.
  • Consistency in Accuracy: Visibility reports were generally accurate (often >90%) regardless of whether the subsequent action was right or wrong.
  • Baseline Matters: Without the visibility question, most GPT models chose the hidden block in less than 7% of cases. The "standard" evaluation protocol was actually making the models look much worse at ToM than they actually are.
Figure 3: Shares of the answered MOVE replies that choose the hidden block, with 95% intervals over scene resamples: without the visibility question, with the block-count question in its place where this batch asks it (gpt-5.6-terra and Qwen3.8-27B were asked it in the batch of Table 2), and with the visibility question, the base prompt. gpt-6-astra, asked only with the visibility question, chooses the hidden block in 2 of 800 replies and is not shown. Counts: Table 6.

Limitations and Future Work

  • Single Paradigm: The study focused exclusively on the Director Task with block labels. It remains to be seen if these findings generalize to other ToM tasks like False Belief tests.
  • English Only: The prompts were exclusively in English.
  • Reasoning Effort: Models were tested at "medium" reasoning effort; higher-compute reasoning might show different dynamics.
  • Human Comparison: While the paper notes that humans also make egocentric errors in the Director Task, it does not test whether asking humans the visibility question first would similarly change their error rates.

The authors conclude that any future ToM benchmark must include a baseline that asks for the action without first eliciting a perspective report to ensure the test isn't measuring its own interference.

Summary updated

Abstract

Vision-language models are now routinely tested on theory of mind: whether they can work out what another person sees, knows, and means. A common design asks the model to report that perspective and then to act for the person, and reads a correct report beside a wrong action as a failure to use the perspective. That reading rests on a quiet assumption: that asking for the report leaves the action unchanged. We show that it does not. In a rendered Director Task, a partition hides the largest block from a director who asks for "the largest block". Asking first which blocks the director can see raises the share of replies that choose this hidden block by 12 to 88 percentage points on six GPT models and by 17 on Qwen3.8-27B, even though the models mostly report correctly which blocks the director sees. Reasoning traces and prompt rewordings suggest why: the question shifts how the request is read, from the director's view toward the model's own. When the prompt states what the director knows and refers to, fewer than 1% of replies pair a correct report with a hidden-block choice. Any evaluation that elicits a perspective report before the action it scores needs a baseline without the question, and we deliver one: paired prompts that differ only in that question, a design any perspective-taking benchmark can adopt.
BibTeX
@misc{guo2026asking,
  author = {Jiaxuan Guo and Xingyu Ren and Kejia Zhang and Jingxin Yang and Youran Sun and Haizhao Yang},
  title = {{Asking What a Director Sees Can Change What Vision-Language Models Choose}},
  year = {2026},
  eprint = {2610.29318},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.29318},
}
Figure