The Embodiment Gap in Vision Language Model Robot Planning

32 concepts24 papers1 proposal

Large Vision-Language Models (VLMs) possess an extraordinary wealth of world knowledge and common sense. They can easily outline the abstract steps required to clean a kitchen spill or organize a cluttered desk. Yet, when these models are deployed directly as robotic controllers or open-loop planners, their performance plummets. This disconnect between high-level semantic capability and low-level physical execution is known as the embodiment gap.

The root of this problem lies in data distribution. As Noriaki Hirose et al. point out in LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Video, the training distributions of general-purpose foundation models are fundamentally misaligned with real-world robotic experiences. Internet text and images lack the precise spatial-temporal grounding required to move a physical arm or navigate a dynamic room safely.

Why General VLMs Fail at Physical Execution

To understand why the embodiment gap exists, we must look at how VLMs process visual information versus what a physical robot actually needs to complete a task.

The Decoupling of Benchmarks and Control

It is tempting to assume that a more capable VLM will automatically make a better robot. However, Jianke Zhang et al. demonstrated a surprising "decoupling" in VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models. They discovered that a VLM’s high score on standard vision-language benchmarks does not reliably predict its success in robotic control. The primary bottleneck is often not the language model's reasoning capacity, but the vision encoder's inability to extract the fine-grained spatial features necessary for physical interaction.

Spatial and Continuous Parameter Blindspots

Robotic manipulation requires reasoning over continuous parameters, such as exact object poses, collision-free trajectories, and contact forces. As Nishanth Kumar et al. observe in Open-World Task and Motion Planning via Vision-Language Model Generated Constraints, while VLMs excel at understanding natural language and common sense, they lack the precision needed to calculate these low-level geometric constraints.

Xian Fu et al. confirmed this limitation in What can VLMs Do for Zero-shot Embodied Task Planning?, showing that even state-of-the-art models like GPT-4V achieve low accuracy in zero-shot embodied planning due to fundamental failures in spatial perception and object type recognition.

The Failure of Standard Prompting

In pure text or vision domains, techniques like Chain-of-Thought (CoT), Self-Verification, and Self-Consistency significantly boost a model's reasoning capabilities. But in robotics, Shogo Akiyama et al. found in Open-loop VLM Robot Planning: An Investigation of Fine-tuning and Prompt Engineering Strategies that these standard prompting strategies often fail to improve—and can even degrade—a VLM's performance on robotic planning benchmarks. This suggests that the reasoning required for physical interaction cannot be unlocked by standard linguistic scaffolding alone.

Quantifying the Performance Deficit

To expose the depth of this gap, researchers have developed rigorous benchmarks that evaluate VLMs in complex, long-horizon, or counterfactual scenarios. The results consistently reveal a massive performance drop compared to human capabilities.

Bridging the Gap via Hierarchical Decoupling

If end-to-end control from a single VLM fails, how can we leverage their semantic power? The emerging consensus across the field is hierarchical decoupling: using the VLM as a high-level semantic reasoner that outputs intermediate abstractions, which are then executed by robust, low-level controllers or specialized policies.

[ High-Level VLM ] ---> Outputs Abstractions ---> [ Low-Level Controller ] ---> Physical Action
(Semantic Reasoning)     (Affordances/Primitives)    (Spatial/Geometric)

Affordances and Constraints as Intermediates

Instead of asking a VLM to output raw motor actions, successful architectures use them to predict geometric landmarks or constraints. In RT-Affordance: Reasoning about Robotic Manipulation with Affordances, Soroush Nasiriany et al. bridge the gap by training models to predict intermediate end-effector poses at key task stages. This structural guidance yields over 50% performance gains compared to standard end-to-end methods. Similarly, the OWL-TAMP framework by Kumar et al. uses VLMs to generate task-specific constraints that guide classical motion planners, combining open-world semantic understanding with mathematically precise execution.

Primitive Skill Sequencing

Another powerful strategy is to ground the VLM's output into a discrete library of pre-trained robotic primitives. Laura Smith et al. introduced STEER: Bridging VLMs and Low-Level Control for Adaptable Robotic Manipulation, which re-annotates existing robot datasets with dense language descriptions of primitive skills. This allows the high-level planner to flexibly compose actions without needing to understand the nuances of expert human control during execution.

To scale this approach, Zeren Chen et al. curated RH20T-P: A Primitive-Level Robotic Manipulation Dataset towards Composable Generalization Agents in Real-world Scenarios, providing a massive corpus of fine-grained primitive annotations specifically designed to help agents learn composable skills that overcome spatial grounding limitations.

Alternative Representations and Symmetries

To address the fact that standard behavior cloning ignores the goal-reaching nature of robotics, Yang Zhang et al. proposed PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations. By using contrastive learning, their system gives models a quantitative understanding of task progress and goal reachability, which drastically improves long-horizon execution.

For contact-rich manipulation where human demonstrations are unavailable, Berk Cicek et al. introduced CoRAL in Contact-VLA: Zero-Shot Planning and Control for Contact-Rich Manipulation, decoupling high-level reasoning from low-level control to provide explainable and adaptable execution. Finally, Mingxi Jia et al. leveraged inherent geometric symmetries in Learning Efficient and Robust Language-Conditioned Manipulation Using Textual-Visual Relevancy and Equivariant Language Mapping, allowing pre-trained VLMs to generalize to unseen manipulation scenarios without the massive data overhead typically required to bridge the embodiment gap.

Key Foundational Work

What can VLMs Do for Zero-shot Embodied Task Planning? (2024) — contributed 3 concept elements to this direction.
ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities (2024) — contributed 2 concept elements to this direction.
RT-Affordance: Reasoning about Robotic Manipulation with Affordances (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_vlm_procedural_planning_limitations_e7758e3b,
  title = {VLM procedural planning limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vlm-procedural-planning-limitations/rdr_e7758e3b69864ce3bb40395cd85f662c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_vlm_procedural_planning_limitations_e7758e3b_v1,
  title = {VLM procedural planning limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vlm-procedural-planning-limitations/rdr_e7758e3b69864ce3bb40395cd85f662c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake