TL;DR
RoboLongR is an offline-to-online framework that teaches robots to master long-horizon tasks by learning "causally consistent" rewards. By combining history-dependent recurrent memory with a novel stage-shuffle supervision technique, the system ensures that robots are rewarded only when they complete subtasks in the correct order, preventing them from "cheating" through visual shortcuts.
What Problem Does This Paper Address?
In robotic manipulation, "long-horizon" tasks—like opening a drawer, placing an object inside, and closing it—are notoriously difficult. Most current methods use visual rewards to guide a robot's learning. However, these rewards often fall into a trap: they reward the appearance of progress rather than the fulfillment of prerequisites.
Consider the drawer example: a closed, empty drawer looks almost identical to a closed, full drawer. If a robot simply closes an empty drawer, a standard visual reward model might give it a high score for "completing" the task, even though it skipped the crucial middle step. The authors call the alignment between rewarded progress and satisfied prerequisites causal consistency. Without it, robots often learn policies that fail to generalize because they haven't truly mastered the underlying task logic.
Methods
RoboLongR tackles this by learning a history-dependent reward model through a four-stage pipeline:
- Causally Consistent Reward Learning: The core innovation is stage-shuffle supervision. The authors take successful demonstrations and partition them into segments. They then create "negative" examples by shuffling the order of these segments (e.g., Open $\rightarrow$ Close $\rightarrow$ Place). The reward model is trained to assign zero reward to these out-of-order sequences. To process this history, the model uses a frozen DINOv2 encoder paired with an autoregressive Transformer and a recurrent memory module.
- Demonstration Stage Annotation: Demos are labeled with coarse semantic stages (e.g., "reach," "grasp," "place") to provide a shared vocabulary for policy conditioning.
- Stage-Conditioned Initialization: A policy is initialized via Behavior Cloning (BC) using these coarse labels.
- Stage-Scheduled Online RL: The robot refines its policy through Reinforcement Learning (PPO). Crucially, RoboLongR includes an Online Reward Correction loop: if the robot repeatedly fails a task (sparse terminal feedback) but the reward model gives it a high score, the system triggers a reward update to penalize those specific failure trajectories.
Datasets
The model was pretrained on a massive multi-task dataset:
- RLBench: Demonstrations from 88 different tasks were used for reward pretraining.
- Open-X Embodiment: Data from the Bridge and Fractal subsets provided additional progress supervision and "rewound" negatives.
- Evaluation: The system was tested on 18 long-horizon RLBench tasks that were strictly excluded from the reward pretraining phase, as well as four real-robot tasks using a Franka Panda arm.
Key Insights and Ideas
- History is a Prerequisite for Causal Rewards: You cannot determine if a task is done correctly just by looking at a single frame. Recurrent memory is essential to distinguish between a "goal-like" state reached correctly versus one reached by skipping steps.
- Synthetic Negatives are Powerful: By manually shuffling demonstration segments, the authors create a "counterfactual" training signal that teaches the model exactly what a prerequisite violation looks like, something rarely found in successful expert demonstrations.
- Two-Round Refinement: The authors found that using a first round of training with uniform segments to "warm up" the model, followed by a second round using adaptive segments based on the model's own predictions, significantly improved reward accuracy.
- Closing the Loop with Sparse Feedback: Even a great reward model will have errors. Using the environment's binary "success/failure" signal to trigger reward corrections during online RL prevents the policy from exploiting reward "holes."
Key Results
RoboLongR demonstrated a significant performance leap over existing baselines like ReWiND:
- Simulation (RLBench): On 18 held-out tasks, RoboLongR achieved a 31.8% average success rate, outperforming the strongest baseline by 9.2 percentage points. It was particularly effective on "strong-dependency" tasks where order is non-negotiable.
- Real-World Robotics: In tests on a Franka Panda, RoboLongR achieved a 57.5% success rate across four complex tasks (Drawer Storage, Board Wiping, Tabletop Tidying, and Arch Bridge Construction). This represented a massive 32.5-percentage-point improvement over the comparison method.
- Diagnostic Accuracy: In "reward-level" tests, RoboLongR achieved 90% ranking accuracy in distinguishing correct trajectories from order-perturbed ones, compared to just 55% for the baseline.
Limitations and Future Work
While RoboLongR pushes the state-of-the-art, the authors note several areas for improvement:
- Supervision Artifacts: Synthetic shuffles can sometimes introduce visual "boundary cues" that the model might overfit to.
- Blunt Correction: The current online correction method penalizes the entire trajectory. Future work could aim to localize the specific error point to provide more granular feedback.
- Reward Hacking: The "progress return" objective can sometimes favor robots that stay in high-reward states for a long time rather than finishing the task quickly.
- Complex Partial Orders: While the model handles some flexible subtask orders, more complex prerequisite graphs (where many valid paths exist) remain a challenge for future research.
Summary updated