Large Vision-Language Models (VLMs) possess an extraordinary wealth of world knowledge and common sense. They can easily outline the abstract steps required to clean a kitchen spill or organize a cluttered desk. Yet, when these models are deployed directly as robotic controllers or open-loop planners, their performance plummets. This disconnect between high-level semantic capability and low-level physical execution is known as the embodiment gap.
The root of this problem lies in data distribution. As Noriaki Hirose et al. point out in LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Video, the training distributions of general-purpose foundation models are fundamentally misaligned with real-world robotic experiences. Internet text and images lack the precise spatial-temporal grounding required to move a physical arm or navigate a dynamic room safely.
Why General VLMs Fail at Physical Execution
To understand why the embodiment gap exists, we must look at how VLMs process visual information versus what a physical robot actually needs to complete a task.
The Decoupling of Benchmarks and Control
It is tempting to assume that a more capable VLM will automatically make a better robot. However, Jianke Zhang et al. demonstrated a surprising "decoupling" in VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models. They discovered that a VLM’s high score on standard vision-language benchmarks does not reliably predict its success in robotic control. The primary bottleneck is often not the language model's reasoning capacity, but the vision encoder's inability to extract the fine-grained spatial features necessary for physical interaction.
Spatial and Continuous Parameter Blindspots
Robotic manipulation requires reasoning over continuous parameters, such as exact object poses, collision-free trajectories, and contact forces. As Nishanth Kumar et al. observe in Open-World Task and Motion Planning via Vision-Language Model Generated Constraints, while VLMs excel at understanding natural language and common sense, they lack the precision needed to calculate these low-level geometric constraints.
Xian Fu et al. confirmed this limitation in What can VLMs Do for Zero-shot Embodied Task Planning?, showing that even state-of-the-art models like GPT-4V achieve low accuracy in zero-shot embodied planning due to fundamental failures in spatial perception and object type recognition.
The Failure of Standard Prompting
In pure text or vision domains, techniques like Chain-of-Thought (CoT), Self-Verification, and Self-Consistency significantly boost a model's reasoning capabilities. But in robotics, Shogo Akiyama et al. found in Open-loop VLM Robot Planning: An Investigation of Fine-tuning and Prompt Engineering Strategies that these standard prompting strategies often fail to improve—and can even degrade—a VLM's performance on robotic planning benchmarks. This suggests that the reasoning required for physical interaction cannot be unlocked by standard linguistic scaffolding alone.
Quantifying the Performance Deficit
To expose the depth of this gap, researchers have developed rigorous benchmarks that evaluate VLMs in complex, long-horizon, or counterfactual scenarios. The results consistently reveal a massive performance drop compared to human capabilities.
- Household Procedural Planning: In ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities, Ying Su et al. showed that current VLMs struggle to generate correct, commonsensical procedural plans over extended action sequences when forced to reason across multi-modal inputs.
- Long-Horizon Manipulation: Shiduo Zhang et al. introduced VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks, finding that state-of-the-art Vision-Language-Action (VLA) models suffer from low success rates when tasks demand intricate planning, generalization, or understanding implicit human intent.
- Autonomous Exploration and Safety: The deficit becomes even more stark during free exploration. In HomeSafeBench: A Benchmark for Embodied Vision-Language Models in Free-Exploration Home Safety Inspection, Jiashu Yao et al. demonstrated that top-tier proprietary models achieve an F1-score of just 10.23% when identifying household hazards. This underscores a critical need for better spatial planning and purposeful navigation, echoing earlier findings by Yuankai Qi et al. in REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments.
Bridging the Gap via Hierarchical Decoupling
If end-to-end control from a single VLM fails, how can we leverage their semantic power? The emerging consensus across the field is hierarchical decoupling: using the VLM as a high-level semantic reasoner that outputs intermediate abstractions, which are then executed by robust, low-level controllers or specialized policies.
[ High-Level VLM ] ---> Outputs Abstractions ---> [ Low-Level Controller ] ---> Physical Action
(Semantic Reasoning) (Affordances/Primitives) (Spatial/Geometric)
Affordances and Constraints as Intermediates
Instead of asking a VLM to output raw motor actions, successful architectures use them to predict geometric landmarks or constraints. In RT-Affordance: Reasoning about Robotic Manipulation with Affordances, Soroush Nasiriany et al. bridge the gap by training models to predict intermediate end-effector poses at key task stages. This structural guidance yields over 50% performance gains compared to standard end-to-end methods. Similarly, the OWL-TAMP framework by Kumar et al. uses VLMs to generate task-specific constraints that guide classical motion planners, combining open-world semantic understanding with mathematically precise execution.
Primitive Skill Sequencing
Another powerful strategy is to ground the VLM's output into a discrete library of pre-trained robotic primitives. Laura Smith et al. introduced STEER: Bridging VLMs and Low-Level Control for Adaptable Robotic Manipulation, which re-annotates existing robot datasets with dense language descriptions of primitive skills. This allows the high-level planner to flexibly compose actions without needing to understand the nuances of expert human control during execution.
To scale this approach, Zeren Chen et al. curated RH20T-P: A Primitive-Level Robotic Manipulation Dataset towards Composable Generalization Agents in Real-world Scenarios, providing a massive corpus of fine-grained primitive annotations specifically designed to help agents learn composable skills that overcome spatial grounding limitations.
Alternative Representations and Symmetries
To address the fact that standard behavior cloning ignores the goal-reaching nature of robotics, Yang Zhang et al. proposed PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations. By using contrastive learning, their system gives models a quantitative understanding of task progress and goal reachability, which drastically improves long-horizon execution.
For contact-rich manipulation where human demonstrations are unavailable, Berk Cicek et al. introduced CoRAL in Contact-VLA: Zero-Shot Planning and Control for Contact-Rich Manipulation, decoupling high-level reasoning from low-level control to provide explainable and adaptable execution. Finally, Mingxi Jia et al. leveraged inherent geometric symmetries in Learning Efficient and Robust Language-Conditioned Manipulation Using Textual-Visual Relevancy and Equivariant Language Mapping, allowing pre-trained VLMs to generalize to unseen manipulation scenarios without the massive data overhead typically required to bridge the embodiment gap.
Key Foundational Work
What can VLMs Do for Zero-shot Embodied Task Planning? (2024) — contributed 3 concept elements to this direction.
ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities (2024) — contributed 2 concept elements to this direction.
RT-Affordance: Reasoning about Robotic Manipulation with Affordances (2024) — contributed 1 concept elements to this direction.