TL;DR
Event2Reward introduces a novel way to train robotic reward models by leveraging "progress relations"—relative comparisons of task completion found in both successful and failed robot executions. By combining absolute temporal labels from expert demonstrations with local constraints that account for stalls and regressions, the system provides dense, accurate rewards that significantly accelerate reinforcement learning (RL) on both simulated and real-world robots.
What Problem Does This Paper Address?
Reinforcement learning for robotics often struggles with "sparse rewards," where a robot only receives feedback at the very end of a task (e.g., a 1 for success, 0 for failure). To speed up learning, researchers use reward shaping, providing a "dense" reward at every step based on task progress.
The standard approach is to use normalized time from expert demonstrations as a proxy for progress. However, this assumes progress is always linear and monotonic. In reality, during RL exploration, robots frequently stall, drop objects (regression), or try again (recovery). A time-based reward model might mistakenly assign a high reward to a state simply because it occurred late in an episode, even if the robot just dropped the tool it was supposed to use. This "misleading positive shaping" credits the very actions that caused the failure, hindering the learning process.
Methods
The authors propose Event2Reward, a task-conditioned progress model built on a Vision-Language Model (VLM) backbone (Qwen3.5-4B). The core innovation lies in its dual-supervision strategy:
- Absolute Progress Supervision: Uses normalized time from successful demonstrations to anchor the numerical scale of progress (0.0 at start, 1.0 at completion).
- Relative Progress Supervision: Uses "event-grounded" relations to constrain the model. For example, if a robot drops a banana, the model is taught that the state after the drop has strictly less progress than the state before the drop, even if the absolute progress value is unknown.
The model predicts a scalar progress value $p_t$, which is then used for Potential-Based Reward Shaping (PBRS). The shaping reward is calculated as the discounted difference between successive progress predictions: $r^{shaped}_t = r^{env}_t + \gamma p_{t+1} - p_t$. This ensures that if progress regresses, the robot receives a negative shaping reward, discouraging the behavior.
Datasets
The study utilizes a massive amount of data across different stages:
- Pretraining: ~20,000 hours of public robot demonstrations (e.g., DROID, RoboMIND) and private human manipulation videos.
- Event-Annotated Dataset: The authors curated a new dataset from ~100K raw real-robot records. They used a two-pass VLM annotation process (GPT-5.6 Sol) followed by human review to identify four event types: progress stall, regression and recovery, no further progress, and unrecovered regression.
- Benchmarks: Evaluation was conducted on 40 tasks from the LIBERO simulation suites and six real-world tasks using Franka robots, including precise insertions (Plug, RAM, Ethernet) and object manipulation (Stack Bowls, Press Buttons, Plate Vegetables).
Key Insights and Ideas
- Relative > Absolute for Failures: It is notoriously difficult to assign an exact "percentage complete" to a failed state. However, it is intuitive to say a state is "better" or "worse" than another. Event2Reward exploits this by using hinge losses to enforce these relative rankings.
- Complementary Supervision: Ablation studies showed that using only relative relations leads to "scale collapse" (where the model predicts tiny, near-zero values), while using only temporal labels leads to poor local accuracy during failures. Joint supervision is essential.
- VLMs as Reward Functions: By using a VLM, the model can incorporate task instructions and visual history, allowing it to distinguish between a robot "waiting" (a necessary stall) and a robot "stuck" (a progress stall).
Key Results
Event2Reward consistently outperformed state-of-the-art baselines like Robometer and Robo-Dopamine.
- Simulation (LIBERO): Improved average policy success by 18.16 percentage points over sparse-reward baselines across 40 tasks.
- Real-World Performance: After just 30 minutes of training, Event2Reward achieved a 60.0% success rate on precise insertion tasks (outperforming Robometer's 37.8%) and 90.0% on manipulation tasks (vs. Robometer's 81.7%).
- Learning Efficiency: In a plug-insertion task, Event2Reward reached 100% success in just 15 minutes, whereas the sparse-reward baseline required 45 minutes.
Limitations and Future Work
- Annotation Effort: While the authors used VLMs to assist, the initial event-level annotation still requires significant effort compared to purely self-supervised methods.
- Inference Latency: Large VLMs like Qwen-4B can be computationally expensive to run in a real-time RL loop, though the authors mitigated this by using action-chunking.
- Future Directions: The authors suggest exploring how these progress relations could be learned autonomously without any human-in-the-loop annotation, perhaps through curiosity-driven exploration or automated failure analysis.
Summary updated