TL;DR
"Seeing What Matters" is a new benchmark that shifts the focus of robotic spatial reasoning from general scene description to "actionable" belief—the ability to find the specific visual evidence required to execute a command. The study reveals a massive performance gap, with state-of-the-art vision-language models (VLMs) succeeding only 26–43% of the time compared to nearly 99% for a task-conditioned reference pipeline.
What Problem Does This Paper Address?
Imagine a robot at a coffee station. To "tamp the coffee," it needs to see the portafilter basket and tamper alignment. To "place the cup," it needs to see the spout and the drip tray. The physical scene is identical, but the information required for success is completely different.
Current spatial reasoning benchmarks often ask models to describe a scene or answer general questions (e.g., "Where is the cup?"). However, for real-world manipulation, "seeing more" isn't always better. A robot needs to acquire the particular evidence that makes a specific command executable. This paper addresses the lack of evaluation for task-conditioned actionable spatial belief: can an agent proactively look for the right information based on what it's trying to do?
Methods
The authors introduce a benchmark and a diagnostic reference pipeline to isolate the "information acquisition" phase from low-level control.
- The Benchmark (SWM): It uses a "frozen RGB" setup where an agent is given a command and must select a limited number of views from a set of 12 candidate images. These views are then passed to a fixed, scripted skill for execution. Success is measured by whether the task actually succeeds in the simulator (hidden postconditions).
- Spatial-Functional Belief Ledger: A structured internal state that tracks task-relevant predicates like part visibility, relative pose, and functional cues (e.g., is the hopper full?).
- View Belief Map (VBM): A rule-based scoring system that ranks candidate views by their expected "evidence gain"—how much new, task-relevant information they are likely to provide for the current command.
- Action-Induced Belief Revision (AIBR): An optional "repair" mechanism that, if a task fails, compares predicted vs. observed outcomes to identify stale information and request a targeted re-observation.
Datasets
The Seeing What Matters (SWM) benchmark consists of 860 frozen evaluation cases across three simulated tabletop domains:
- Coffee Workstation: 500 cases covering tamping, insertion, placement, serving, and grinding.
- Drawer/Cabinet: 180 cases focusing on handle access and reachability.
- Microwave/Appliance: 180 cases focusing on cavity support and control states.
The benchmark includes over 115,000 evaluator episodes, including a "functional out-of-distribution" (OOD) split where the objects remain the same but the task requirements change, forcing the model to follow the command rather than relying on object-level shortcuts.
Key Insights and Ideas
- Actionable belief is task-conditioned: A camera view that is perfect for one task (e.g., seeing the grinder controls) can be totally useless for another (e.g., checking cup alignment), even in the same scene.
- The "Binding" Gap: The primary reason VLMs fail isn't a lack of visual recognition; it's an inability to "bind" the current command to the specific functional evidence chain needed for execution.
- Seeing more != Seeing better: Geometrically accurate global beliefs can still fail if they omit the one specific relation (like "is the cup under the spout?") required for the next action.
- Functional evidence matters most when shortcuts fail: In the functional-OOD tests, models that relied on object category shortcuts (e.g., "it's a coffee machine, so look at the front") failed completely when the task required specific internal details.
Key Results
- Reference Pipeline Dominance: The metadata-assisted reference pipeline achieved 98.8% success in closed-loop tasks and 98.9% in functional-OOD tasks.
- VLM Struggle: Seven state-of-the-art VLMs (including models like Grok 4.3 and Llama 4 Maverick) reached only 26%–43% success. While they often detected "target-related" cues, they failed to consistently select the evidence that actually supported the task outcome.
- Ablation Collapse: Removing the "command binding" component (making the view selection task-agnostic) caused success rates to plummet to near zero in complex tasks.
- Image Content Dependency: Replacing simulator metadata with RGB-derived scores (using a CLIP-based head) preserved high success (98.5%), proving that the benchmark measures information contained in the images themselves.
Limitations and Future Work
- Simulation vs. Reality: The benchmark is currently limited to simulated MuJoCo environments. Real-world deployment would introduce noise and more complex lighting/occlusion.
- Scripted Skills: The evaluation uses fixed scripted skills rather than learned low-level controllers. While this isolates the "perception" problem, it doesn't account for how perception and control might be more tightly coupled in end-to-end systems.
- Scale: While the coffee domain is detailed, the other two domains (drawers and microwaves) are currently used only as small-scale sanity checks.
- Future Directions: The authors suggest that future work should focus on developing learned view-selection policies that can match the "command-to-evidence" logic of their reference pipeline without needing simulator metadata.
Summary updated