TL;DR
Researchers have found that the "brain fog" Vision-Language Models (VLMs) experience in complex planning tasks isn't necessarily a failure of logic, but an entanglement of perception and state estimation. By narrowing the VLM's role to simple local perception and delegating multi-view reconciliation to a constraint solver, the new VLM-as-perceiver paradigm boosts plan success rates from near-zero to over 90% on challenging, occluded environments.
What Problem Does This Paper Address?
In visual planning, a model must look at a scene, understand the state of objects, and generate a sequence of actions to reach a goal. Currently, VLMs are "entangled": they try to perceive objects, reconcile different camera angles, and calculate a plan all in one go.
This leads to two major failure modes:
- Depth Ambiguity: In a single view, it’s often impossible to tell if Block A is resting on Block B or on a hidden Block C behind it. VLMs struggle to "stitch" multiple views together to resolve these contradictions.
- Scene Density: As the number of objects grows (scaling from 6 blocks to 100), VLMs simply stop "seeing" everything. They don't necessarily hallucinate; they just fail to report the full inventory, making a valid plan impossible to generate.
Existing benchmarks are often too simple (usually capped at 6 blocks) to expose these bottlenecks, leading to a false sense of "saturation" in the field.
Methods
The authors propose a shift in architecture called VLM-as-perceiver. They compare three paradigms:
- VLM-as-planner: The model receives images and outputs a plan directly. (High failure rate).
- VLM-as-formalizer: The model translates images into a symbolic language (like PDDL), which a classical solver then handles. (Fails at multi-view reconciliation).
- VLM-as-perceiver (Proposed): The VLM is restricted to describing what it sees in one view at a time.
- Parser: Decomposes the VLM's local descriptions into "relational claims" (e.g., "Blue is on Red").
- Reconciler: Uses a MaxSMT solver (Z3) to weigh these claims. It treats the scene as a logic puzzle, finding the most mathematically likely global state that satisfies physical constraints (e.g., two blocks can't occupy the same space).
- Planner: Feeds the reconciled state into a classical planner (Fast Downward) to generate the final action sequence.
Datasets
The paper introduces BW-PO (Blocksworld under Partial Observability), a benchmark designed to break standard VLM planners. It consists of two subsets:
- BW-PO-DEPTH: Specifically designed with occlusions that create depth ambiguity, requiring multi-view integration to solve.
- BW-PO-N: Scales the number of blocks ($N$) from 6 up to 100 to test how models handle increasing scene density.
Key Insights and Ideas
- Reconciliation is a Logic Problem, Not a Vision Problem: The "aha moment" here is that VLMs are actually decent at local perception but terrible at the "bookkeeping" required to merge conflicting views. By moving reconciliation to a deterministic solver, the researchers "solved" the depth ambiguity problem.
- The Recall Bottleneck: At high object counts (100 blocks), the bottleneck isn't that the models are "stupid"—it's that they have low recall. Open-weight models like Gemma only recovered about 10% of the blocks in dense scenes. If the model doesn't see the block, it can't plan for it.
- Stochastic vs. Deterministic: The paper argues for moving as much of the pipeline as possible from stochastic (probabilistic VLM outputs) to deterministic (logic solvers) components to improve reliability and error localization.
Key Results
The performance jumps were massive when switching to the "perceiver" approach:
- Gemma-4-31B: Plan success soared from 1% to 97% on depth-ambiguity tasks.
- Qwen3.5-122B: Jumped from 0% to 95% on the same tasks.
- Gemini-3.1-Pro: Improved from 3% to 84% on the most challenging 100-block scenes.
- Efficiency: VLM-as-perceiver is significantly cheaper, reducing the number of generated tokens by a median of 1.8x, and up to 44x in the most complex scenarios.
Limitations and Future Work
- Domain Specificity: The current reconciler is tailored to the "vertical stack" logic of Blocksworld. Applying this to more chaotic, non-discrete environments (like a messy kitchen) would require a much more complex constraint model.
- Object Identity: The study assumes the model knows the full inventory of objects beforehand. In a real-world "open-world" scenario, discovering new objects while planning remains a major challenge.
- Resolution Limits: As the number of blocks increases, the number of pixels per block decreases. Future work needs to disentangle whether models fail at high $N$ because of "cognitive load" or simply because the image resolution is too low to read object identifiers.
Summary updated