RoboLongR: Causally Consistent Rewards Teach Robots to Progressively Master Long-Horizon Manipulation

Zixuan Chen, Lin Cao, Fenggang Zhang, Haotong Zhang, Yangtao Chen, Jieqi Shi, Jing Huo, Yang GaoarXiv 2026 · arXiv

TL;DR

RoboLongR is an offline-to-online framework that teaches robots to master long-horizon tasks by learning "causally consistent" rewards. By combining history-dependent recurrent memory with a novel stage-shuffle supervision technique, the system ensures that robots are rewarded only when they complete subtasks in the correct order, preventing them from "cheating" through visual shortcuts.

What Problem Does This Paper Address?

In robotic manipulation, "long-horizon" tasks—like opening a drawer, placing an object inside, and closing it—are notoriously difficult. Most current methods use visual rewards to guide a robot's learning. However, these rewards often fall into a trap: they reward the appearance of progress rather than the fulfillment of prerequisites.

Consider the drawer example: a closed, empty drawer looks almost identical to a closed, full drawer. If a robot simply closes an empty drawer, a standard visual reward model might give it a high score for "completing" the task, even though it skipped the crucial middle step. The authors call the alignment between rewarded progress and satisfied prerequisites causal consistency. Without it, robots often learn policies that fail to generalize because they haven't truly mastered the underlying task logic.

The "visual progress trap": closing an empty drawer looks like success but is actually a failure due to skipped prerequisites.

Methods

RoboLongR tackles this by learning a history-dependent reward model through a four-stage pipeline:

  1. Causally Consistent Reward Learning: The core innovation is stage-shuffle supervision. The authors take successful demonstrations and partition them into segments. They then create "negative" examples by shuffling the order of these segments (e.g., Open $\rightarrow$ Close $\rightarrow$ Place). The reward model is trained to assign zero reward to these out-of-order sequences. To process this history, the model uses a frozen DINOv2 encoder paired with an autoregressive Transformer and a recurrent memory module.
  2. Demonstration Stage Annotation: Demos are labeled with coarse semantic stages (e.g., "reach," "grasp," "place") to provide a shared vocabulary for policy conditioning.
  3. Stage-Conditioned Initialization: A policy is initialized via Behavior Cloning (BC) using these coarse labels.
  4. Stage-Scheduled Online RL: The robot refines its policy through Reinforcement Learning (PPO). Crucially, RoboLongR includes an Online Reward Correction loop: if the robot repeatedly fails a task (sparse terminal feedback) but the reward model gives it a high score, the system triggers a reward update to penalize those specific failure trajectories.
The RoboLongR framework, showing the transition from offline reward learning to online policy refinement and correction.

Datasets

The model was pretrained on a massive multi-task dataset:

  • RLBench: Demonstrations from 88 different tasks were used for reward pretraining.
  • Open-X Embodiment: Data from the Bridge and Fractal subsets provided additional progress supervision and "rewound" negatives.
  • Evaluation: The system was tested on 18 long-horizon RLBench tasks that were strictly excluded from the reward pretraining phase, as well as four real-robot tasks using a Franka Panda arm.
The 18 long-horizon simulation tasks used to evaluate zero-shot reward transfer and policy adaptation.

Key Insights and Ideas

  • History is a Prerequisite for Causal Rewards: You cannot determine if a task is done correctly just by looking at a single frame. Recurrent memory is essential to distinguish between a "goal-like" state reached correctly versus one reached by skipping steps.
  • Synthetic Negatives are Powerful: By manually shuffling demonstration segments, the authors create a "counterfactual" training signal that teaches the model exactly what a prerequisite violation looks like, something rarely found in successful expert demonstrations.
  • Two-Round Refinement: The authors found that using a first round of training with uniform segments to "warm up" the model, followed by a second round using adaptive segments based on the model's own predictions, significantly improved reward accuracy.
  • Closing the Loop with Sparse Feedback: Even a great reward model will have errors. Using the environment's binary "success/failure" signal to trigger reward corrections during online RL prevents the policy from exploiting reward "holes."

Key Results

RoboLongR demonstrated a significant performance leap over existing baselines like ReWiND:

  • Simulation (RLBench): On 18 held-out tasks, RoboLongR achieved a 31.8% average success rate, outperforming the strongest baseline by 9.2 percentage points. It was particularly effective on "strong-dependency" tasks where order is non-negotiable.
  • Real-World Robotics: In tests on a Franka Panda, RoboLongR achieved a 57.5% success rate across four complex tasks (Drawer Storage, Board Wiping, Tabletop Tidying, and Arch Bridge Construction). This represented a massive 32.5-percentage-point improvement over the comparison method.
  • Diagnostic Accuracy: In "reward-level" tests, RoboLongR achieved 90% ranking accuracy in distinguishing correct trajectories from order-perturbed ones, compared to just 55% for the baseline.

Limitations and Future Work

While RoboLongR pushes the state-of-the-art, the authors note several areas for improvement:

  • Supervision Artifacts: Synthetic shuffles can sometimes introduce visual "boundary cues" that the model might overfit to.
  • Blunt Correction: The current online correction method penalizes the entire trajectory. Future work could aim to localize the specific error point to provide more granular feedback.
  • Reward Hacking: The "progress return" objective can sometimes favor robots that stay in high-reward states for a long time rather than finishing the task quickly.
  • Complex Partial Orders: While the model handles some flexible subtask orders, more complex prerequisite graphs (where many valid paths exist) remain a challenge for future research.

Summary updated

Abstract

Long-horizon manipulation requires actions whose validity depends on earlier interactions, yet visual progress can conceal missing task prerequisites. We propose RoboLongR, an offline-to-online framework that learns history-dependent rewards to guide policy adaptation to new tasks. Its central objective is causal consistency: aligning rewarded progress with prerequisites supported by execution history. Stage-shuffle supervision encourages execution-order sensitivity, recurrent memory preserves evidence of earlier interactions, and two-round training refines trajectory segments using predicted progress. The resulting rewards guide stage-conditioned policy refinement from behavior cloning to online reinforcement learning. Sparse terminal feedback connects policy experience to reward learning by triggering correction for repeated high-scoring failures. Pretrained on demonstrations from 88 RLBench tasks and Open-X data, RoboLongR achieves 31.8% average success on 18 long-horizon tasks excluded from reward pretraining, exceeding the strongest evaluated baseline by 9.2 percentage points. Across four real-robot tasks, it achieves 57.5% success, a 32.5-percentage-point improvement over the comparison method. Reward-level controls probe sensitivity to synthetic order perturbations, while ablations examine the contributions of individual components to policy improvement.
BibTeX
@misc{chen2026robolongr,
  author = {Zixuan Chen and Lin Cao and Fenggang Zhang and Haotong Zhang and Yangtao Chen and Jieqi Shi and Jing Huo and Yang Gao},
  title = {{RoboLongR: Causally Consistent Rewards Teach Robots to Progressively Master Long-Horizon Manipulation}},
  year = {2026},
  eprint = {2610.33863},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.33863},
}
Figure