Grounding High Level Reasoning in Low Level Motor Control

605 concepts515 papers4 proposals

Large Language Models (LLMs) and Vision-Language Models (VLMs) possess an extraordinary grasp of world knowledge, semantic relationships, and high-level logic. They can easily decompose an abstract human request like "clean up the spilled coffee" into a logical sequence of subgoals: locate a towel, move to the counter, wipe the surface.

Yet, when these models are deployed on physical robots, a profound disconnect emerges. A robot's hardware does not understand "wipe the counter"; it operates on high-frequency, continuous streams of Cartesian coordinates, joint velocities, and motor torques. This disconnect is known as the reasoning-execution gap or the embodiment gap.

Bridging this gap requires translating discrete, qualitative semantic concepts into continuous, quantitative physical actions without losing the generalization capabilities of large-scale models.

The Representation Mismatch

Why can't we simply prompt a powerful language model to output raw motor commands? The core issue lies in a fundamental mismatch between the modalities of language and physical dynamics.

As argued by Aravind Rajeswaran et al. in From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Control, natural language is inherently too imprecise and lossy to capture the nuances of contact-rich physical tasks or high-precision motor control. When a model attempts to map language directly to joint movements, the resulting trajectories are often noisy and brittle. Kevin Black et al. observed in Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models that while large models excel at broad semantic reasoning, they lack the fine-grained spatial coordination required for localized physical interactions like grasping.

Furthermore, forcing a generalist model to output raw numerical control parameters can degrade its cognitive capabilities. To solve this, modern research focuses on designing intermediate representations that act as a translation layer between the "brain" (the semantic model) and the "brawn" (the low-level controller).

Strategy 1: Action Co-Tokenization

One elegant way to bridge the gap is to treat robot actions exactly like text tokens. If a model can predict the next word in a sentence, it can theoretically predict the next motor command in a trajectory.

[Text Tokens] ---> "Pick up the yellow sponge" ---> [VLA Model] ---> [Action Tokens] ---> [Robot Actuators]

In the seminal work RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control, Brianna Zitkovich et al. demonstrated that by representing robotic trajectories as discrete text tokens, massive vision-language models can be co-fine-tuned directly on physical actions. This unified approach allows the model's web-scale semantic knowledge to guide low-level control directly, enabling emergent behaviors like grasping an object based on abstract concepts (e.g., picking up an item that can be used as a hammer).

However, relying solely on discrete tokens can struggle with high-frequency, continuous physical feedback. To mitigate this, Jianke Zhang et al. developed UniCoD: Enhancing Robot Policy via Unified Continuous and Discrete Representation Learning, which unifies discrete language tokens with continuous visual representations. This hybrid approach ensures the robot maintains strong linguistic grounding while remaining sensitive to real-time physical dynamics.

Strategy 2: Hierarchical Translation and Latent Bridges

Instead of forcing a single end-to-end model to handle both abstract strategy and motor execution, a second major paradigm introduces hierarchical translation layers. This concept builds on classical robotics principles; for instance, Dana H. Ballard originally proposed Task Frames in Robot Manipulation to attach coordinate systems directly to manipulated objects, simplifying complex trajectories into invariant geometric commands.

Modern frameworks implement this hierarchy by introducing intermediate vocabularies. In RT-H: Action Hierarchies using Language, Suneel Belkhale et al. split the control loop by predicting intermediate "language motions" (such as "move arm forward") between the high-level task and low-level actions. This creates a flexible grammar of movement that allows humans or higher-level planners to intervene and correct the robot using natural language.

[High-Level Goal] ---> "Clean table"
         |
         v
[Language Motion] ---> "Move arm forward"
         |
         v
[Low-Level Action] --> [Continuous Torques / Velocities]

Alternatively, the bridge does not have to be linguistic. The aforementioned framework by Aravind Rajeswaran et al. uses specialized neural "latent codes" rather than text strings to connect LLMs to low-level controllers. These continuous latent vectors act as a non-lossy interface, preserving the rich spatial and physical nuances that get dropped when translating intent into pure text.

Strategy 3: Coupling Autoregression with Diffusion

A third powerful intuition addresses the temporal mismatch between reasoning and execution. High-level planning is slow and reflective (autoregressive), while physical control must be fast and reactive.

To combine these strengths, researchers have begun pairing autoregressive foundation models with diffusion policies. Diffusion models excel at generating smooth, continuous, and multi-modal trajectories by iteratively denoising action sequences.

In DiffusionVLA: Scaling Robot Foundation Models via Unified Diffusion and Autoregression, Junjie Wen et al. designed a system where an autoregressive VLM generates high-level semantic reasoning, which is then injected directly into a diffusion-based action generation loop. This architecture allows the robot to maintain a deep cognitive understanding of the task while executing precise, reactive physical control at real-time speeds (up to 82Hz).

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_to_physical_grounding_09180908,
  title = {semantic-to-physical grounding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-to-physical-grounding/rdr_09180908635442159b0d2b0ea980c048},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_to_physical_grounding_09180908_v1,
  title = {semantic-to-physical grounding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-to-physical-grounding/rdr_09180908635442159b0d2b0ea980c048/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake