Bridging the Physical Grounding Gap in LLM Planning

474 concepts368 papers8 proposals

Imagine asking a state-of-the-art Large Language Model (LLM) to help you clean a kitchen. It will effortlessly generate a beautifully structured, step-by-step plan: *1. Pick up the mug from the table.

  1. Open the cupboard.
  2. Place the mug inside.* On paper, this plan is flawless. But hand this plan to a physical robot, and it will likely fail on step one because its hands are already full, or fail on step three because it tries to phase the mug directly through a closed cupboard door.

This disconnect is known as the physical grounding gap (or semantic-physical misalignment). LLMs possess immense semantic "book smarts"—they understand how words relate to other words—but they lack "street smarts"—an inherent understanding of physical laws, spatial geometry, and real-world constraints. To build truly autonomous embodied agents, researchers are developing mechanisms to anchor abstract linguistic reasoning into concrete physical reality.

Why the Gap Exists: Semantics vs. Physics

The root of the problem lies in the nature of text-based training data. LLMs learn from human language, where common-sense physical steps are rarely documented due to reporting bias. As highlighted by Manling Li et al. in Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making, humans omit "obvious" preconditions in text, such as needing to open a container before retrieving its contents. When an LLM plans, it frequently skips these crucial physical prerequisites because they are linguistically implicit.

Furthermore, text lacks the fine-grained spatial and kinematic constraints of the real world. A plan might be linguistically probable but physically impossible due to environmental geometry, friction, or hardware limits. For instance, Lin Guan et al. demonstrate in Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning that LLMs consistently fail to infer implicit spatial constraints, such as the necessity of removing a top object before accessing one beneath it. This leads to "plan hallucinations"—the generation of actions that violate the fundamental physics of the environment.

Closing the Gap: Core Mechanisms

To bridge this gap, researchers have moved away from pure "zero-shot" text generation, instead introducing architectures that force language models to interact with, or be constrained by, physical realities.

1. Affordance Gating (The "SayCan" Paradigm)

One of the most intuitive solutions is to filter what an LLM wants to do by what the robot can actually do in that exact moment. In Do As I Can, Not As I Say: Grounding Language in Robotic Affordances, Brian Ichter et al. introduced a framework that evaluates two distinct probabilities for any given action:

  1. Semantic Probability ("Say"): How likely is this action to help achieve the high-level goal, according to the LLM?
  2. Feasibility Probability ("Can"): What is the actual probability that the robot's low-level controllers can successfully execute this action given the current state of the environment?

By multiplying these two scores, the system ensures that the robot never attempts a semantically logical but physically impossible action, such as trying to grab an object that is out of reach or attempting to hold an item when its manipulators are already occupied.

2. Neuro-Symbolic Integration and Classical Planners

Rather than forcing an LLM to compute exact physical trajectories, another prominent approach uses the LLM as a high-level translator that hands off the actual execution logic to a provably sound classical planner.

Historically, classical automated planning relied on formal languages like PDDL (Planning Domain Definition Language) to map out strict physical preconditions and effects, a challenge explored early on by Siddharth Srivastava et al. in Metaphysics of Planning Domain Descriptions. Modern frameworks combine these worlds. For example, Alejandro Agostini in Bootstrapping Object-level Planning with Large Language Models leverages LLMs to generate high-level object-level subgoals, which are then parsed into formal PDDL constraints. A traditional Task and Motion Planning (TAMP) solver then handles the low-level geometry, ensuring that the final plan strictly respects physical laws.

Similarly, Joonhyung Lee et al. in SPOTS: Stable Placement of Objects with Reasoning in Semi-Autonomous Teleoperation Systems decouple semantic reasoning from physical verification. The LLM decides where an object contextually belongs (e.g., a plate goes on the dining table), while a physics-based simulator verifies the exact 3D spatial coordinates to ensure physical stability and prevent collisions.

3. Interactive Simulation and Error-Correction Loops

If an LLM lacks an intuitive understanding of physical cause and effect, one direct solution is to let it experience the physical world through simulation and learn from its mistakes.

In Language Models Meet World Models: Embodied Experiences Enhance Language Models, Jiannan Xiang et al. demonstrate that fine-tuning language models on interactive experiences within virtual environments significantly improves their understanding of object permanence and spatial consistency. Instead of relying solely on static text, the model learns the direct causal consequences of its proposed actions by seeing how they transform the environment.

When errors do occur during execution, closed-loop feedback allows the agent to self-correct dynamically. Hanlin Wang et al. in E²CL: Exploration-based Error Correction Learning for Embodied Agents propose an exploration-based learning framework where agents intentionally interact with environments to trigger and learn from physical errors. This enables a capability called speculative inference, where the model anticipates physical constraints and corrects its plan before execution.

Conclusion

The physical grounding gap highlights that linguistic fluency is not equivalent to physical understanding. As A. K. Kovalev notes in Application of Pretrained Large Language Models in Embodied Artificial Intelligence, without explicit grounding mechanisms or "reality regularizers," LLMs will continue to propose actions that violate physical laws. By combining the semantic breadth of language models with the mathematical rigor of affordance gating, classical planners, and interactive simulations, modern robotics is successfully turning abstract text into reliable, real-world action.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_physical_grounding_gap_dc08c7f8,
  title = {physical grounding gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/physical-grounding-gap/rdr_dc08c7f89d9a4765a37fa06118920d97},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_physical_grounding_gap_dc08c7f8_v1,
  title = {physical grounding gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/physical-grounding-gap/rdr_dc08c7f89d9a4765a37fa06118920d97/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake