Automated Reward Synthesis for Scalable Robot Learning

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 21,266 papers1,334 conceptsfrozen 2026-10-04

In traditional reinforcement learning (RL), the "reward function" is the bridge between human intent and machine action. To teach a robot to open a drawer, an engineer might write a mathematical formula that rewards the hand for getting closer to the handle and the handle for moving along its track.

However, this process—known as reward engineering—is notoriously brittle. As noted by Bohan Wu and Chelsea Finn in their work on EMBR, manually defining these functions for a library of diverse skills is labor-intensive and prone to human error. If the reward is slightly off, the robot might "hack" the metric—optimizing the numerical score through unnatural or unintended physical behaviors, a problem highlighted by Irfan Essa and colleagues in Words into Action.

To build generalist robots, researchers are moving away from hand-coded formulas toward automated reward synthesis. This direction treats the reward function not as a fixed input from a human, but as a learnable or generative component of the AI system itself.

Large Language Models as Reward Architects

The most recent shift in this field leverages the semantic reasoning of Large Language Models (LLMs) to bridge the gap between high-level instructions and low-level control. Instead of a human writing Python code to define "success," an LLM can "see" a task and write the reward code itself.

In Agentic Skill Discovery (ASD), Xufeng Zhao and Stefan Wermter demonstrate how LLMs can autonomously propose a curriculum of skills and verify success using visual feedback. Similarly, the SDS (See it. Do it. Sorted.) framework by Jeffrey Li uses GPT-4o to analyze a single video demonstration of a quadruped and automatically generate executable reward functions. This eliminates the need for motion capture or manual trial-and-error, allowing complex gaits to emerge from a single visual prompt.

Learning from Visual Success and Play

Another approach to bypassing reward engineering is to use visual data as the ground truth for success. If a robot can recognize what a "finished" task looks like, it can generate its own internal reward signal based on how closely its current state matches that goal.

  • Reward Sketching: Researchers at OpenAI and DeepMind have explored reward sketching, where humans provide high-level progress curves over videos, which the system then uses to label massive amounts of offline robot experience.
  • Visual Goal-Conditioning: Frameworks like Actionable Models by Alex Irpan and Karol Hausman allow robots to learn from unlabeled, offline datasets. By treating every state in a dataset as a potential "goal," the robot learns a functional distance metric—essentially learning how to reach any visual state it has seen before without a single human-defined reward.
  • Play Data: Corey Lynch and Ted Xiao showed in Play-LMP that robots can learn more robustly from unlabelled "play" data—continuous logs of human interaction—than from narrow, task-specific demonstrations.

Self-Play and Unsupervised Discovery

When a robot has no human guidance at all, it must discover its own rewards through curiosity or competition. This is often framed as unsupervised skill discovery.

In Asymmetric Self-Play, OpenAI researchers set up a "challenge and response" game between two agents, Alice and Bob. Alice's "reward" is to find a configuration that Bob cannot replicate, while Bob's reward is to successfully mimic Alice. This creates an automated curriculum where the difficulty of the "reward" naturally scales with the robot's ability.

For locomotion, Boyuan Liang and Lingfeng Sun demonstrated that robots can autonomously discover energy-efficient gaits by simply minimizing a velocity-dependent energy term. This "energy-centric" reward replaces the need for meticulously designed footfall sequences, allowing the robot to "find" the walk or trot that best fits its own physics.

The Shift to Generalist Agents

The common thread across these papers is the realization that the bottleneck in robotics is no longer just the learning algorithm, but the data labeling. By automating reward synthesis—whether through LLM-generated code, visual goal-matching, or self-play—researchers are enabling robots to learn from the vast, unlabelled "dark matter" of human videos and robot play data.

As Yilun Du and colleagues argue in UniPi, casting robot learning as a generative problem (like text-to-video) allows us to leverage internet-scale data. This moves the field away from specialized, brittle controllers and toward generalist agents capable of mastering hundreds of tasks without a single line of hand-crafted reward code.

Papers in this version

Showing 60 of 1,266 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 1360 papers
  2. Version 1 · 2026-10-04 · 1266 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_manual_reward_engineering_dab6f4ff,
  title = {manual reward engineering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manual-reward-engineering/rdr_dab6f4ff243843ac8725262b8a06b542},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_automated_reward_synthesis_for_scalable__dab6f4ff_v1,
  title = {Automated Reward Synthesis for Scalable Robot Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-reward-synthesis-for-scalable-robot-learning/rdr_dab6f4ff243843ac8725262b8a06b542/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}