Seeing What Matters: Benchmarking Task-Conditioned Actionable Spatial Belief for Manipulation

Yina Jian, Mingkun Xu

TL;DR

"Seeing What Matters" is a new benchmark that shifts the focus of robotic spatial reasoning from general scene description to "actionable" belief—the ability to find the specific visual evidence required to execute a command. The study reveals a massive performance gap, with state-of-the-art vision-language models (VLMs) succeeding only 26–43% of the time compared to nearly 99% for a task-conditioned reference pipeline.

What Problem Does This Paper Address?

Imagine a robot at a coffee station. To "tamp the coffee," it needs to see the portafilter basket and tamper alignment. To "place the cup," it needs to see the spout and the drip tray. The physical scene is identical, but the information required for success is completely different.

Current spatial reasoning benchmarks often ask models to describe a scene or answer general questions (e.g., "Where is the cup?"). However, for real-world manipulation, "seeing more" isn't always better. A robot needs to acquire the particular evidence that makes a specific command executable. This paper addresses the lack of evaluation for task-conditioned actionable spatial belief: can an agent proactively look for the right information based on what it's trying to do?

Figure 2. From spatial belief to actionable evidence. Theory-of-Space-style evaluation asks whether an agent can construct and revise spatial beliefs through exploration. Seeing What Matters keeps the active-belief motivation but scores whether command-conditioned views expose the evidence needed by a shared scripted skill, with hidden postconditions used only for evaluation. Schematic modules are illustrative.

Methods

The authors introduce a benchmark and a diagnostic reference pipeline to isolate the "information acquisition" phase from low-level control.

  • The Benchmark (SWM): It uses a "frozen RGB" setup where an agent is given a command and must select a limited number of views from a set of 12 candidate images. These views are then passed to a fixed, scripted skill for execution. Success is measured by whether the task actually succeeds in the simulator (hidden postconditions).
  • Spatial-Functional Belief Ledger: A structured internal state that tracks task-relevant predicates like part visibility, relative pose, and functional cues (e.g., is the hopper full?).
  • View Belief Map (VBM): A rule-based scoring system that ranks candidate views by their expected "evidence gain"—how much new, task-relevant information they are likely to provide for the current command.
  • Action-Induced Belief Revision (AIBR): An optional "repair" mechanism that, if a task fails, compares predicted vs. observed outcomes to identify stale information and request a targeted re-observation.
The reference pipeline: from command to actionable evidence via a View Belief Map and Belief Ledger.

Datasets

The Seeing What Matters (SWM) benchmark consists of 860 frozen evaluation cases across three simulated tabletop domains:

  1. Coffee Workstation: 500 cases covering tamping, insertion, placement, serving, and grinding.
  2. Drawer/Cabinet: 180 cases focusing on handle access and reachability.
  3. Microwave/Appliance: 180 cases focusing on cavity support and control states.

The benchmark includes over 115,000 evaluator episodes, including a "functional out-of-distribution" (OOD) split where the objects remain the same but the task requirements change, forcing the model to follow the command rather than relying on object-level shortcuts.

Key Insights and Ideas

  • Actionable belief is task-conditioned: A camera view that is perfect for one task (e.g., seeing the grinder controls) can be totally useless for another (e.g., checking cup alignment), even in the same scene.
  • The "Binding" Gap: The primary reason VLMs fail isn't a lack of visual recognition; it's an inability to "bind" the current command to the specific functional evidence chain needed for execution.
  • Seeing more != Seeing better: Geometrically accurate global beliefs can still fail if they omit the one specific relation (like "is the cup under the spout?") required for the next action.
  • Functional evidence matters most when shortcuts fail: In the functional-OOD tests, models that relied on object category shortcuts (e.g., "it's a coffee machine, so look at the front") failed completely when the task required specific internal details.
Different commands require different evidence even within the same scene.

Key Results

  • Reference Pipeline Dominance: The metadata-assisted reference pipeline achieved 98.8% success in closed-loop tasks and 98.9% in functional-OOD tasks.
  • VLM Struggle: Seven state-of-the-art VLMs (including models like Grok 4.3 and Llama 4 Maverick) reached only 26%–43% success. While they often detected "target-related" cues, they failed to consistently select the evidence that actually supported the task outcome.
  • Ablation Collapse: Removing the "command binding" component (making the view selection task-agnostic) caused success rates to plummet to near zero in complex tasks.
  • Image Content Dependency: Replacing simulator metadata with RGB-derived scores (using a CLIP-based head) preserved high success (98.5%), proving that the benchmark measures information contained in the images themselves.
Figure 1. Seven-model evidence diagnostics. Left: success versus target-cue rate. Right: success versus wrong-command cue rate. Seven strict-clean VLMs each have 1,500 evaluation records, pre- view twelve RGB candidates, and select exactly three views. Dots show point estimates; icons label models. Cue rates are separate evaluator diagnostics, not complementary fractions.

Limitations and Future Work

  • Simulation vs. Reality: The benchmark is currently limited to simulated MuJoCo environments. Real-world deployment would introduce noise and more complex lighting/occlusion.
  • Scripted Skills: The evaluation uses fixed scripted skills rather than learned low-level controllers. While this isolates the "perception" problem, it doesn't account for how perception and control might be more tightly coupled in end-to-end systems.
  • Scale: While the coffee domain is detailed, the other two domains (drawers and microwaves) are currently used only as small-scale sanity checks.
  • Future Directions: The authors suggest that future work should focus on developing learned view-selection policies that can match the "command-to-evidence" logic of their reference pipeline without needing simulator metadata.

Summary updated

Abstract

When should a robot look at the cup, and when at the portafilter? Spatial-reasoning benchmarks evaluate whether a model can describe a scene, but manipulation needs something more specific: the ability to acquire the particular evidence that makes the current command executable. A view that is informative for one task can be useless for another, even at the same workstation. We introduce Seeing What Matters, a benchmark for task-conditioned actionable spatial belief in simulated tabletop manipulation. Across three domains--coffee workstation, drawer/cabinet, and microwave/appliance--the benchmark provides 860 frozen evaluation cases; expanding across methods, budgets, and inference repeats yields more than 115,000 evaluator episodes under two protocols: closed-loop actionability and same-object functional out-of-distribution generalization. Each case pairs a set of candidate RGB views with a manipulation command, shared scripted skills, and hidden postconditions, so the benchmark directly measures whether a view-selection policy acquires the observations its command actually needs. We also provide a diagnostic reference pipeline that maintains a spatial-functional belief state and ranks views by missing evidence. At primary budget settings, this pipeline reaches 98.8% closed-loop and 98.9% functional-OOD success; ablating the functional or command-binding components collapses performance to near zero. Seven vision-language models, evaluated on the same cases, reach only 26-43%. Replacing the reference pipeline's coverage metadata with frozen RGB-derived scores preserves 98.5%/98.9% success; shuffling the visual scores collapses it to 15%/11%, confirming that the result depends on image content. These gaps suggest that spatial belief for manipulation should be evaluated not by scene-level completeness, but by whether it acquires the evidence the current command requires.
BibTeX
@misc{jian2026seeing,
  author = {Yina Jian and Mingkun Xu},
  title = {{Seeing What Matters: Benchmarking Task-Conditioned Actionable Spatial Belief for Manipulation}},
  year = {2026},
  eprint = {2611.09794},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09794},
}
Figure