Event2Reward: Learning Robotic Rewards from Progress Relations

Junnan Nie, Zitian Fan, Jiachen Zhang, Junyi Lao, Likui Zhang, Chenghao Liu, Xiaoyang Lu, Yunbo Wang, Yixue Zhang, Linbo Zhai, Jiayi Li, Mengzhe Wang, Xichang Wu, Bin Wang, Chang Liu, Tianle Zhang, Songfang HuangarXiv 2026 · arXiv

TL;DR

Event2Reward introduces a novel way to train robotic reward models by leveraging "progress relations"—relative comparisons of task completion found in both successful and failed robot executions. By combining absolute temporal labels from expert demonstrations with local constraints that account for stalls and regressions, the system provides dense, accurate rewards that significantly accelerate reinforcement learning (RL) on both simulated and real-world robots.

What Problem Does This Paper Address?

Reinforcement learning for robotics often struggles with "sparse rewards," where a robot only receives feedback at the very end of a task (e.g., a 1 for success, 0 for failure). To speed up learning, researchers use reward shaping, providing a "dense" reward at every step based on task progress.

The standard approach is to use normalized time from expert demonstrations as a proxy for progress. However, this assumes progress is always linear and monotonic. In reality, during RL exploration, robots frequently stall, drop objects (regression), or try again (recovery). A time-based reward model might mistakenly assign a high reward to a state simply because it occurred late in an episode, even if the robot just dropped the tool it was supposed to use. This "misleading positive shaping" credits the very actions that caused the failure, hindering the learning process.

Methods

The authors propose Event2Reward, a task-conditioned progress model built on a Vision-Language Model (VLM) backbone (Qwen3.5-4B). The core innovation lies in its dual-supervision strategy:

  1. Absolute Progress Supervision: Uses normalized time from successful demonstrations to anchor the numerical scale of progress (0.0 at start, 1.0 at completion).
  2. Relative Progress Supervision: Uses "event-grounded" relations to constrain the model. For example, if a robot drops a banana, the model is taught that the state after the drop has strictly less progress than the state before the drop, even if the absolute progress value is unknown.
Figure 2: Overview of progress supervision in Event2Reward. Left: normalized temporal labels from successful expert trajectories anchor the progress scale of a shared vision-language model. Right: execution events in robot rollouts support relative constraints, illustrated by a drop and subsequent recovery (p8 > p12 and p16 > p12). These complementary signals supervise the same progress model; relation supervision uses both successful and failed executions.

The model predicts a scalar progress value $p_t$, which is then used for Potential-Based Reward Shaping (PBRS). The shaping reward is calculated as the discounted difference between successive progress predictions: $r^{shaped}_t = r^{env}_t + \gamma p_{t+1} - p_t$. This ensures that if progress regresses, the robot receives a negative shaping reward, discouraging the behavior.

Datasets

The study utilizes a massive amount of data across different stages:

  • Pretraining: ~20,000 hours of public robot demonstrations (e.g., DROID, RoboMIND) and private human manipulation videos.
  • Event-Annotated Dataset: The authors curated a new dataset from ~100K raw real-robot records. They used a two-pass VLM annotation process (GPT-5.6 Sol) followed by human review to identify four event types: progress stall, regression and recovery, no further progress, and unrecovered regression.
  • Benchmarks: Evaluation was conducted on 40 tasks from the LIBERO simulation suites and six real-world tasks using Franka robots, including precise insertions (Plug, RAM, Ethernet) and object manipulation (Stack Bowls, Press Buttons, Plate Vegetables).

Key Insights and Ideas

  • Relative > Absolute for Failures: It is notoriously difficult to assign an exact "percentage complete" to a failed state. However, it is intuitive to say a state is "better" or "worse" than another. Event2Reward exploits this by using hinge losses to enforce these relative rankings.
  • Complementary Supervision: Ablation studies showed that using only relative relations leads to "scale collapse" (where the model predicts tiny, near-zero values), while using only temporal labels leads to poor local accuracy during failures. Joint supervision is essential.
  • VLMs as Reward Functions: By using a VLM, the model can incorporate task instructions and visual history, allowing it to distinguish between a robot "waiting" (a necessary stall) and a robot "stuck" (a progress stall).

Key Results

Event2Reward consistently outperformed state-of-the-art baselines like Robometer and Robo-Dopamine.

  • Simulation (LIBERO): Improved average policy success by 18.16 percentage points over sparse-reward baselines across 40 tasks.
  • Real-World Performance: After just 30 minutes of training, Event2Reward achieved a 60.0% success rate on precise insertion tasks (outperforming Robometer's 37.8%) and 90.0% on manipulation tasks (vs. Robometer's 81.7%).
  • Learning Efficiency: In a plug-insertion task, Event2Reward reached 100% success in just 15 minutes, whereas the sparse-reward baseline required 45 minutes.
Figure 1: Overview of Event2Reward. Left: pretraining data and real robot rollouts containing both successful and failed executions. Task events provide within-trajectory progress relations that complement temporal labels from successful demonstrations. Top center: task-conditioned progress predictions support reward shaping. Right: six real-world downstream tasks. The performance panels summarize real-world and LIBERO policy success, plug-insertion learning time, and the effect of rollout-based joint training. Plug-insertion time is measured until a rolling window of 10 autonomous training episodes reaches 100% success.

Limitations and Future Work

  • Annotation Effort: While the authors used VLMs to assist, the initial event-level annotation still requires significant effort compared to purely self-supervised methods.
  • Inference Latency: Large VLMs like Qwen-4B can be computationally expensive to run in a real-time RL loop, though the authors mitigated this by using action-chunking.
  • Future Directions: The authors suggest exploring how these progress relations could be learned autonomously without any human-in-the-loop annotation, perhaps through curiosity-driven exploration or automated failure analysis.

Summary updated

Abstract

Learning task progress from robot demonstrations provides a practical way to construct dense shaping rewards for robotic reinforcement learning. A common approach uses normalized demonstration time as progress labels, implicitly assuming monotonically increasing task progress. During policy exploration, however, actual task progress can stall or even regress. Such models may mistake regressions for progress, assigning positive shaping rewards to actions that cause progress loss. Real suboptimal and failed executions offer corrective experience, but assigning absolute progress values to their states is difficult. Our key observation is that stalls, regressions, and recoveries establish local relative progress relations even when absolute progress is unknown. We introduce Event2Reward, which uses normalized temporal labels to anchor the progress scale and these within-trajectory relations to constrain local changes, combining absolute and relative progress supervision. Its progress predictions serve as potentials for reward shaping. We curate an event-annotated dataset from approximately 100K raw real-robot execution records, covering successful, suboptimal, and failed executions, without requiring framewise absolute progress labels. Under the same training budgets, Event2Reward improves average policy success by 18.16 percentage points over a baseline using only sparse task rewards across 40 tasks in four LIBERO suites. On real robots, average success after 30 minutes of wall-clock training reaches 60.0% on three precise insertion tasks and 90.0% on three object manipulation tasks, exceeding the sparse-reward baseline under the same budget by 36.7 and 23.3 percentage points, respectively. Ablation studies further show that absolute progress supervision and relative progress relations are complementary, with joint supervision yielding better downstream policy learning.
BibTeX
@misc{nie2026event,
  author = {Junnan Nie and Zitian Fan and Jiachen Zhang and Junyi Lao and Likui Zhang and Chenghao Liu and Xiaoyang Lu and Yunbo Wang and Yixue Zhang and Linbo Zhai and Jiayi Li and Mengzhe Wang and Xichang Wu and Bin Wang and Chang Liu and Tianle Zhang and Songfang Huang},
  title = {{Event2Reward: Learning Robotic Rewards from Progress Relations}},
  year = {2026},
  eprint = {2610.39212},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39212},
}
Figure