TL;DR
This paper provides the first comprehensive "semantics-first" taxonomy and benchmark for Robotic Reward Models (RRMs)—the foundation-model-based evaluators replacing manual reward engineering. By introducing RRM-Bench-1K, a dataset of 1,117 real-robot executions with dense task-state labels, the authors reveal a critical "chronology trap": most current models reward elapsed time and motion rather than actual task fulfillment.
What Problem Does This Paper Address?
In the era of generalist robots, manually writing a reward function for every new task is a bottleneck. Researchers have turned to Robotic Reward Models (RRMs)—evaluators that use foundation models (like CLIP or GPT-4o) to judge robot behavior based on a natural language instruction or a video demonstration.
However, the field lacks a unified understanding of what these models actually "reward." Some predict success, others estimate progress, and some compare two videos to see which is better. Furthermore, existing benchmarks often use "agreement with frame order" as a metric. This is a flawed test: a model that simply rewards the passage of time would pass this test, even if the robot is just waving its arm in the air without completing the task. The authors set out to build a benchmark that forces models to distinguish between superficial task-like motion and genuine task success.
Methods
The authors contribute three major components to the field:
- A Semantic Taxonomy: They organized 114 existing RRM methods into five "families" based on the question their output answers: Outcome (Is it done?), Absolute Progress (How far along is it?), Relative (Which is better?), Temporal Value (How much time/reward is left?), and Hybrid.
- RRM-Bench-1K: A new benchmark designed to expose "proxy failures." It uses four "contrast suites":
- Action–Effect: Does the model reward the motion of wiping a table even if the stain remains?
- Semantic/Ordered Constraints: Does it reward putting an object in the wrong box?
- Persistence/Reversal: Does the reward drop if the robot drops an object it just picked up?
- Visibility: Does the model know when it can't see the object it's supposed to judge?
- RRM-Hub: An open-source toolbox that standardizes the interface for 14 specialized RRMs and several general-purpose Vision-Language Models (VLMs), allowing for reproducible comparisons.
Datasets
- RRM-Bench-1K (New): Contains 1,117 real-robot videos across 52 tasks and 5 different robot embodiments (including ALOHA and Franka). It is uniquely annotated with dense task-state labels (tracking exactly when conditions are met or lost) and camera-evidence labels (noting if a specific view is sufficient to judge the task).
- The survey also reviews the data foundations of the 114 methods, including teleoperation corpora (Open X-Embodiment), human internet videos, and simulated rollouts.
Key Insights and Ideas
- The Chronology Trap: Many models correlate more strongly with "elapsed time" than with the actual "task state." If a robot stalls, these models often keep increasing the reward simply because the video is continuing.
- Endpoint vs. Dense Credit: There is a gap between a model's ability to pick the best final video (endpoint selection) and its ability to provide a useful training signal during the execution (dense credit). Models like RoboReward are great at filtering successful trials but poor at detecting when a robot loses progress mid-task.
- Broken Constraints are Hard: Detecting a "wrong destination" (e.g., putting a tool in the wrong drawer) is the hardest challenge for current models. They often perceive any movement toward a container as progress, regardless of the instruction's constraints.
- The "Abstention" Problem: No current RRM effectively signals when it lacks the visual evidence to make a judgment. When an object is hidden from view, models tend to guess or report "no progress" rather than admitting they can't see the state.
Key Results
- Top Performers: General-purpose VLMs (like GPT-4o and Claude 3.5) and specialized models (like Flora-Rewarder) are excellent at ranking whole executions, picking the most complete one in over 75% of cases.
- Dense Tracking Failures: On the "Persistence/Reversal" suite, most models failed to reliably withdraw credit when a task condition was undone (e.g., an object falling off a shelf).
- VLM Query Styles: The way you ask a VLM matters. "Pair-style" queries (comparing two frames) tend to recall more "falls" in progress but suffer from more false alarms compared to "Sequence-style" queries (viewing the whole video).
- Multiview Advantage: Using multiple synchronized cameras significantly improves performance, but the results are highly sensitive to which camera is treated as the "primary" view.
Limitations and Future Work
- Real-Failure Supervision: Most models are trained on successful demonstrations. The authors argue that RRMs need more exposure to real robot failures to learn the boundaries of task success.
- Beyond Instructions: Natural language instructions are often too vague. Future work should explore structured task specifications (like sub-goals or post-conditions) to help models verify specific constraints.
- Abstention and Calibration: There is a dire need for models that can output a "confidence" or "insufficient evidence" signal when the decisive state is not visible.
- Metric Reform: The authors suggest that future RRM research should move away from simple correlation with time and instead use metrics that specifically test for credit withdrawal during regressions.
Summary updated