Synthesizing Robotic Behaviors through Intermediate Representations

This direction is split into other directions: fixed action primitives. Its last version stays readable and citable here.

Version 1 of 1203 papers211 conceptsfrozen 2026-10-04

Traditional robotics operates on a "library" model. A developer programs a set of discrete primitives—like pick, place, or open_door—and a high-level planner sequences them to solve a task. While reliable, this approach hits a fundamental bottleneck: the robot is only as capable as its pre-defined vocabulary. If a task requires a nuanced movement not in the library, such as a "moonwalk" or a specific "poking" motion to clear clutter, the system fails.

Recent research has shifted from selecting skills to synthesizing them. By using Large Language Models (LLMs) and Vision-Language Models (VLMs) to generate intermediate representations—such as reward functions, 3D value maps, or optimization constraints—robots can now manifest novel behaviors on the fly.

The Primitive Bottleneck

The core problem identified across this field is the rigidity of symbolic actions. As noted in Modeling Long-horizon Tasks as Sequential Interaction Landscapes by Soren Pirk and colleagues, systems relying on fixed action symbols cannot autonomously discover new physical interactions. This "closed-vocabulary" constraint prevents robots from adapting to the messy, unpredictable nature of the real world.

Even when high-level reasoning is handled by a powerful LLM, the robot is often tethered to a static API. Yusuke Mikami argues in Natural Language as Polices that bypassing these task-specific APIs in favor of direct coordinate-level control or free-form reasoning allows for much higher success rates in novel scenarios.

Bridging the Gap with Intermediate Representations

The breakthrough lies in finding the right "language" for the LLM to speak to the robot's hardware. Instead of outputting a command like "walk," researchers are asking models to generate the underlying mathematical or spatial structures that define a movement.

1. Reward and Value Map Synthesis

One of the most effective strategies is to have the LLM act as a reward designer. In Language to Rewards for Robotic Skill Synthesis, Wenhao Yu and his team demonstrate that GPT-4 can bridge the gap between human instructions and low-level control by generating reward functions. These functions are then optimized in real-time by a physics engine (like MuJoCo MPC), allowing a quadruped to perform complex, unprogrammed motions.

Similarly, Wenlong Huang and Chen Wang introduced VoxPoser, which uses LLMs to synthesize 3D value maps. Instead of a "pick" command, the model generates a spatial map where the target object has high value and obstacles have low value. A motion planner then finds the optimal path through this "landscape," enabling the robot to handle complex geometries without a single pre-defined skill.

2. Grounding through Affordances

A robot cannot execute a plan if it doesn't know what it is physically capable of doing in its current state. Brian Ichter and Fei Xia addressed this in SayCan by multiplying the probability of an LLM's suggestion ("Say") with a learned affordance model ("Can"). This ensures the synthesized plan is grounded in the robot's actual environment. Grounded Decoding takes this further by guiding the LLM's token generation step-by-step with physical world constraints, ensuring the generated text is physically executable from the start.

3. Optimization and Code Generation

For tasks requiring high precision, such as drone swarms or multi-robot coordination, researchers are using LLMs to generate the mathematical constraints of the problem itself. Carmen Amo Alonso introduces NARRATE, a framework where natural language is translated into free-form optimization problems for a Model Predictive Control (MPC) solver. This allows for high-precision control without task-specific training data.

In the same vein, EmbodiedCoder by Lin Zheng treats the robot as a software engineer, generating real-time code for trajectory synthesis. This enables contact-rich tasks like pouring water or opening doors by tailoring the trajectory to the specific geometry of the object.

Why This Works: The Power of Composition

The intuition behind this shift is that while LLMs are poor at low-level motor control, they are excellent at compositional reasoning. By breaking a task down into a sequence of intermediate representations—whether they are foot contact patterns for locomotion or mark-based visual affordances for manipulation—we leverage the LLM's semantic breadth while relying on classical control and optimization for physical safety and precision.

As Murtaza Dalal shows in Plan-Seq-Learn, this modularity allows robots to solve long-horizon tasks with an 85%+ success rate without ever needing a pre-defined skill library. We are moving away from robots that "know" a few things well, toward robots that can "figure out" how to do almost anything.

Papers in this version

Showing 60 of 203 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_synthesizing_robotic_behaviors_through_i_47b7c226,
  title = {Synthesizing Robotic Behaviors through Intermediate Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthesizing-robotic-behaviors-through-intermediate-representations/rdr_47b7c22680b04295af1a9a15967306ba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_synthesizing_robotic_behaviors_through_i_47b7c226_v1,
  title = {Synthesizing Robotic Behaviors through Intermediate Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthesizing-robotic-behaviors-through-intermediate-representations/rdr_47b7c22680b04295af1a9a15967306ba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}