Traditional robotic systems are notoriously brittle. Training an agent to grasp an object or navigate a room typically requires thousands of trials within a closed-world environment. When faced with a novel object—such as a spatula instead of a ladle—or an implicit instruction like "I am thirsty," these systems fail because they lack an underlying understanding of what these objects are and how they function.
A rapidly growing body of research circumvents this data bottleneck by exploiting a surprising resource: the latent physical common sense embedded within large-scale pre-trained language and vision-language models (VLMs). As noted by Zhiyuan Xu et al. in A Survey on Robotics with Foundation Models: toward Embodied AI, internet-scale pre-training exposes models to vast amounts of human knowledge about physics, geometry, and functional affordances. By grounding this latent knowledge into robotic control loops, researchers are enabling zero-shot generalization across novel objects, complex tasks, and unstructured environments.
The Semantic Bridge for Zero-Shot Generalization
Why can a model trained on internet text and images help a robot manipulate a physical object? The answer lies in semantic connectivity—the ability to map disparate modalities into a shared, continuous embedding space where functional and geometric concepts align.
When a robot encounters an unfamiliar object, it does not need to learn its physical properties from scratch. Instead, it projects the object's visual features into this shared space to identify similarities to known concepts. For instance, Chao Tang 0001 et al. demonstrate in Task-Oriented Grasp Prediction with Visual-Language Inputs that vision-language models allow robots to infer that a bowl can be treated similarly to a cup for pouring tasks based purely on semantic similarity. In GraspGPT: Leveraging Semantic Knowledge from a Large Language Model for Task-Oriented Grasping, the authors show that describing an unfamiliar tool's shape and function in natural language allows a robot to immediately deduce an appropriate grasping strategy based on geometric features it has encountered elsewhere.
[Physical Observation] ----> (Shared Semantic Space) <---- [Human Intent / Text]
|
v
[Zero-Shot Robotic Action]
This semantic bridge also redefines how humans specify goals and constraints. Rather than engineering precise 3D cost functions, researchers can use natural language to reshape a robot's trajectory on the fly. Pratyusha Sharma et al. in Correcting Robot Plans with Natural Language Feedback map verbal feedback (e.g., "stay away from the yellow bottle") directly to a residual cost function that alters the robot's optimization landscape. Because the underlying model understands synonyms and broad semantic associations, the robot generalizes to novel phrasing and unseen entities zero-shot, a capability echoed by Arthur Bucker et al. in Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers.
Furthermore, Amber Xie et al. in Language-Conditioned Path Planning utilize this semantic understanding to specify safety constraints, allowing a robot to understand which objects are safe to bump into in cluttered spaces. Going further, Yuchen Cui et al. demonstrate in Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? that foundation models can bridge the gap between abstract human references—like internet images or hand sketches—and actual robotic observations, serving as an effective zero-shot reward signal for reinforcement learning.
Decomposing Tasks via Affordances and Spatial Constraints
Beyond simple recognition, physical reasoning requires understanding how parts of objects interact over time. Foundation models excel at decomposing long-horizon, complex tasks into manageable semantic sub-goals or geometric constraints without requiring task-specific fine-tuning.
Consider the challenge of task-oriented grasping: picking up a knife to cut an apple requires a completely different grip than picking it up to hand it to a human safely. In DexGANGrasp: Dexterous Generative Adversarial Grasping Synthesis for Task-Oriented Manipulation, the authors leverage multimodal models to reason about object affordances and user intent from open-vocabulary prompts, identifying exactly which part of an irregular geometry should be targeted. Similarly, Yingbo Tang et al. in AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter use the common sense of VLMs to resolve implicit instructions (like "I am thirsty"), mapping the command to the correct object and the specific part to grasp.
To translate this high-level reasoning into low-level motor commands, Haoxu Huang et al. introduce a framework in CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models that decomposes tasks into part-level geometric constraints (e.g., "align the hammer head with the nail"). This allows the robot to resolve complex spatial rotations that traditional point-based methods struggle to handle.
For long-horizon tasks, Yash Shukla et al. in LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agents treat an LLM's output as a directed graph of sub-goals, creating a roadmap that guides reinforcement learning agents through complex sequences without pre-existing skills. This capability enables what Yutao Ouyang et al. describe in Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models as the zero-shot discovery of complex strategies—such as a quadruped robot autonomously figuring out how to build a makeshift staircase to reach a high switch.
Even in specialized domains like surgery, Masoud Moghani et al. show in SuFIA: Language-Guided Augmented Dexterity for Robotic Surgical Assistants that translating verbal commands into high-level Python code allows surgical robots to automate tedious sub-tasks across novel tool geometries. When faced with entirely unseen manipulation tasks, Jiaming Zhou et al. demonstrate in Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization that LLMs can plan actions by selecting relevant in-context examples that share similar underlying physical dynamics.
Preserving Knowledge and Action Tokenization
A critical challenge in this paradigm is the transition from high-level semantic reasoning to low-level physical execution. If a foundation model is fine-tuned directly on narrow robotic trajectory data, it risks suffering from catastrophic forgetting, erasing the very world knowledge that made it useful.
To combat this, recent architectures focus on aligning actions directly with the model's native linguistic or visual representations. Xindi Wu et al. propose a training paradigm in Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting that represents low-level robotic actions as text tokens. This parameter-efficient fine-tuning preserves the model's foundational world knowledge, enabling cross-modal capabilities such as a robot following instructions in a language or cultural context completely absent from its physical demonstration data.
Traditional Fine-Tuning: VLM Weights ---> Overwritten by Trajectories ---> Loss of World Knowledge
Action Tokenization: VLM Weights ---> Retained (Actions as Text) ---> Preserved World Knowledge
Alternatively, researchers can initialize control models directly with pre-trained vision-language weights. As shown by Pengxiang Ding et al. in QUAR-VLA: Vision-Language-Action Model for Quadruped Robots, this initialization transfers semantic reasoning directly to physical tasks, allowing quadrupeds to interpret novel spatial commands and avoid obstacles dynamically.
Instead of predicting raw pixels or joint angles, Jacob Berg et al. frame the world model itself as a visual question-answering problem in Semantic World Models, predicting task-relevant semantic outcomes. This shift allows the robot to plan through abstract concepts (like color, shape, and state changes) rather than getting bogged down in high-dimensional visual noise.
Navigating and Mapping Unstructured Environments
The utility of latent physical reasoning extends beyond manipulation to spatial navigation and mapping. Traditional geometric maps lack semantic understanding; a robot might see a bush or a couch as merely an impassable 3D obstacle. By integrating foundation models, robots can infer the latent physical properties of their surroundings based on general knowledge rather than explicit maps.
In Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models, the authors demonstrate that a quadruped robot can use a VLM to reason over its interaction history and environment, identifying crawlable spaces under furniture or recognizing which vegetation is safe to step through.
Furthermore, Chuhao Liu et al. show in FM-Fusion: Instance-Aware Semantic Mapping Boosted by Vision-Language Foundation Models that leveraging foundation models for 3D semantic mapping eliminates the domain gaps that typically plague supervised detectors. This allows robots to reconcile messy, open-set text labels into consistent, instance-aware 3D maps of entirely novel environments.
Ultimately, this research direction works because it decouples what to do (semantic, functional, and physical logic) from how to do it (low-level torque and trajectory execution). By treating internet-scale foundation models as repositories of common-sense physics, robotics can move away from rigid, closed-world programming and toward flexible, open-vocabulary autonomy.
Key Foundational Work
GraspGPT: Leveraging Semantic Knowledge from a Large Language Model for Task-Oriented Grasping (2023) — contributed 2 concept elements to this direction.
Correcting Robot Plans with Natural Language Feedback (2022) — contributed 1 concept elements to this direction.
Task-Oriented Grasp Prediction with Visual-Language Inputs (2023) — contributed 1 concept elements to this direction.