Robotic Reward Models in the Foundation-Model Era

Guoqiang Liang, Yiqi Lin, Shangchen Zhou, Wenzheng Zeng, Yanzhe Chen, Zechen Bai, Mike Zheng ShouarXiv 2026 · arXiv

TL;DR

This paper provides the first comprehensive "semantics-first" taxonomy and benchmark for Robotic Reward Models (RRMs)—the foundation-model-based evaluators replacing manual reward engineering. By introducing RRM-Bench-1K, a dataset of 1,117 real-robot executions with dense task-state labels, the authors reveal a critical "chronology trap": most current models reward elapsed time and motion rather than actual task fulfillment.

What Problem Does This Paper Address?

In the era of generalist robots, manually writing a reward function for every new task is a bottleneck. Researchers have turned to Robotic Reward Models (RRMs)—evaluators that use foundation models (like CLIP or GPT-4o) to judge robot behavior based on a natural language instruction or a video demonstration.

However, the field lacks a unified understanding of what these models actually "reward." Some predict success, others estimate progress, and some compare two videos to see which is better. Furthermore, existing benchmarks often use "agreement with frame order" as a metric. This is a flawed test: a model that simply rewards the passage of time would pass this test, even if the robot is just waving its arm in the air without completing the task. The authors set out to build a benchmark that forces models to distinguish between superficial task-like motion and genuine task success.

Methods

The authors contribute three major components to the field:

  1. A Semantic Taxonomy: They organized 114 existing RRM methods into five "families" based on the question their output answers: Outcome (Is it done?), Absolute Progress (How far along is it?), Relative (Which is better?), Temporal Value (How much time/reward is left?), and Hybrid.
  2. RRM-Bench-1K: A new benchmark designed to expose "proxy failures." It uses four "contrast suites":
  3. Action–Effect: Does the model reward the motion of wiping a table even if the stain remains?
  4. Semantic/Ordered Constraints: Does it reward putting an object in the wrong box?
  5. Persistence/Reversal: Does the reward drop if the robot drops an object it just picked up?
  6. Visibility: Does the model know when it can't see the object it's supposed to judge?
  7. RRM-Hub: An open-source toolbox that standardizes the interface for 14 specialized RRMs and several general-purpose Vision-Language Models (VLMs), allowing for reproducible comparisons.

Datasets

  • RRM-Bench-1K (New): Contains 1,117 real-robot videos across 52 tasks and 5 different robot embodiments (including ALOHA and Franka). It is uniquely annotated with dense task-state labels (tracking exactly when conditions are met or lost) and camera-evidence labels (noting if a specific view is sufficient to judge the task).
  • The survey also reviews the data foundations of the 114 methods, including teleoperation corpora (Open X-Embodiment), human internet videos, and simulated rollouts.

Key Insights and Ideas

  • The Chronology Trap: Many models correlate more strongly with "elapsed time" than with the actual "task state." If a robot stalls, these models often keep increasing the reward simply because the video is continuing.
  • Endpoint vs. Dense Credit: There is a gap between a model's ability to pick the best final video (endpoint selection) and its ability to provide a useful training signal during the execution (dense credit). Models like RoboReward are great at filtering successful trials but poor at detecting when a robot loses progress mid-task.
  • Broken Constraints are Hard: Detecting a "wrong destination" (e.g., putting a tool in the wrong drawer) is the hardest challenge for current models. They often perceive any movement toward a container as progress, regardless of the instruction's constraints.
  • The "Abstention" Problem: No current RRM effectively signals when it lacks the visual evidence to make a judgment. When an object is hidden from view, models tend to guess or report "no progress" rather than admitting they can't see the state.

Key Results

  • Top Performers: General-purpose VLMs (like GPT-4o and Claude 3.5) and specialized models (like Flora-Rewarder) are excellent at ranking whole executions, picking the most complete one in over 75% of cases.
  • Dense Tracking Failures: On the "Persistence/Reversal" suite, most models failed to reliably withdraw credit when a task condition was undone (e.g., an object falling off a shelf).
  • VLM Query Styles: The way you ask a VLM matters. "Pair-style" queries (comparing two frames) tend to recall more "falls" in progress but suffer from more false alarms compared to "Sequence-style" queries (viewing the whole video).
  • Multiview Advantage: Using multiple synchronized cameras significantly improves performance, but the results are highly sensitive to which camera is treated as the "primary" view.

Limitations and Future Work

  • Real-Failure Supervision: Most models are trained on successful demonstrations. The authors argue that RRMs need more exposure to real robot failures to learn the boundaries of task success.
  • Beyond Instructions: Natural language instructions are often too vague. Future work should explore structured task specifications (like sub-goals or post-conditions) to help models verify specific constraints.
  • Abstention and Calibration: There is a dire need for models that can output a "confidence" or "insufficient evidence" signal when the decisive state is not visible.
  • Metric Reform: The authors suggest that future RRM research should move away from simple correlation with time and instead use metrics that specifically test for credit withdrawal during regressions.

Summary updated

Abstract

Robotic reward models (RRMs) use foundation models to replace per-task reward engineering with reusable, task-conditioned evaluation. Yet under one name, their outputs answer different questions, from whether a task succeeded to which execution is better. This survey organizes 114 methods into five semantic families by the question their native outputs answer: Outcome, Absolute Progress, Relative, Temporal Value, and Hybrid. Within this taxonomy, we review how model design determines what an evaluator can judge, how raw experience becomes supervision, and how existing evaluation practice tests the result. The review exposes a gap: agreement with demonstration order and downstream policy gains do not show whether a reward tracks task fulfillment. We therefore introduce RRM-Bench-1K, 1,117 real-robot videos of closed-loop executions with task-state and camera-evidence labels, and four contrast suites that separate the credited condition from its usual cues. A unified study of 27 evaluators shows that the strongest final scores rank whole executions well enough for selection and filtering. Dense credit, however, follows the progression of an execution more closely than its task state, and no evaluator signals when the decisive view is missing. How much fidelity a reward model needs therefore depends on the decision it drives. We outline directions from real-failure supervision to metrics that separate task state from chronology, and open-source RRM-Hub, a toolbox in which 14 existing RRMs can be readily applied and new evaluators tested on the same capabilities.
BibTeX
@misc{liang2026robotic,
  author = {Guoqiang Liang and Yiqi Lin and Shangchen Zhou and Wenzheng Zeng and Yanzhe Chen and Zechen Bai and Mike Zheng Shou},
  title = {{Robotic Reward Models in the Foundation-Model Era}},
  year = {2026},
  eprint = {2611.20741},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20741},
}
Figure