Rollback: Experience Backtracking in World Models for Policy Improvement

Jiayi Luo, Yang Zhang, Liliang Chen, Nan Wang, Chen Gao, Boxiang Qiu, Zhuoxuan Li, Baorui Peng, Chuankang Li, Yuxiang Yan, Si Liu, Qingyun Sun, Shuicheng Yan, Maoqing Yao, Jianxin Li, Guanghui RenarXiv 2026 · arXiv

TL;DR

Rollback is a new robot learning paradigm that treats world models as "rewindable" environments rather than just forward-only simulators. By backtracking to critical decision points to explore counterfactual futures, it generates high-quality, action-level supervision that allows robots to surpass human performance in complex tasks like tabletop curling and ball tossing within just 8 hours of training.

What Problem Does This Paper Address?

Reinforcement Learning (RL) on physical robots is notoriously difficult because real-world interaction is slow, expensive, and requires constant human resets. While "world models"—AI-powered simulators learned from video—offer a scalable alternative, most current methods use them like a standard video game: you start a run, go forward until you fail, and then start over.

This "forward-only" approach is inefficient. It spends a massive amount of computation on uninformative parts of a trajectory and struggles with credit assignment—figuring out exactly which specific move in a long sequence caused the eventual failure. Furthermore, as a robot's policy improves, it starts visiting states the world model hasn't seen before, leading to "hallucinations" where the simulator's physics no longer match reality.

Methods

The authors introduce Rollback, which turns world models into "counterfactual engines." Instead of just running forward, the system can "rewind" to a specific past state and ask, "What if I had moved the arm slightly to the left here instead?"

Key technical components include:

  • Experience-Backtracking Tree Search: Using Monte Carlo Tree Search (MCTS), the system identifies critical decision points in successful or failed real-world attempts. It then branches out multiple "imagined" futures from those exact points.
  • Tree-Structured Value Estimation: By comparing the outcomes of these sibling branches (which all started from the same state), the system can isolate the effect of a single action chunk. This provides fine-grained supervision without needing to learn a separate, complex "value model."
  • Test-Time Physics Grounding: To prevent the world model from drifting, the system uses a "co-evolution" strategy. Real-world data collected by the robot is used to continuously fine-tune the world model via a lightweight LoRA (Low-Rank Adaptation) pipeline, ensuring the "imagined" physics stay grounded in reality.
The Rollback paradigm compared to standard Real-World RL and Forward-Rollout World Models.

Datasets

The system was evaluated on three challenging real-world manipulation tasks using an AGIBOT Genie-02 robot:

  1. Block Extraction: Removing Jenga blocks without collapsing the tower (requires precise contact physics).
  2. Ball Tossing: Throwing a ball into a bucket (requires precise release timing).
  3. Tabletop Curling: A competitive game where the robot must slide stones near a target while potentially displacing an opponent's stones.

The models were initialized using teleoperated demonstrations but quickly evolved beyond that initial data.

Key Insights and Ideas

  • Rewindability is a Superpower: The ability to "undo" and "reconsider" is a capability that physical reality does not afford. Rollback exploits this unique advantage of learned simulators to concentrate "imagination" where it matters most.
  • Contrastive Supervision: Comparing "what happened" vs. "what could have happened" from the exact same state provides a much cleaner training signal than comparing two entirely different trajectories.
  • Asynchronous Scalability: The authors built a three-tier (Cloud-Edge-Device) infrastructure. This allows the heavy lifting of "imagining" thousands of futures to happen on a GPU cluster in the cloud while the robot continues to act in the real world, exchanging data only when ready.
Overview of the Rollback system, showing the search, value estimation, and co-evolution loops.

Key Results

Rollback demonstrated significant improvements over state-of-the-art baselines like RISE and RECAP:

  • Performance Gains: It improved success rates by an absolute 10.67% to 11.85% across all tasks compared to the strongest baselines.
  • Superhuman Capability: Despite being trained from mediocre human teleoperation data, Rollback surpassed the average success rate of human players in both ball tossing and tabletop curling within 8 hours of training.
  • Stability: Unlike forward-rollout methods that often fluctuate during training, Rollback showed steady, monotonic improvement, largely thanks to the "test-time physics grounding" that kept the world model accurate.
Rollback surpassing human and teleoperation performance over 8 hours of training.

Limitations and Future Work

While impressive, Rollback has a few hurdles to clear:

  • Reward Dependency: It currently relies on a task-specific "world judge" (a reward model) to determine if an imagined rollout succeeded. Future work aims to learn these outcome detectors jointly with the world model.
  • Horizon Limits: Even with grounding, prediction errors eventually accumulate over very long, contact-rich sequences.
  • Single-Task Focus: The current experiments were conducted on single tasks. Scaling this to a general-purpose, multi-task robot policy is the next logical step.

Summary updated

Abstract

Action-conditioned world models offer a scalable alternative to costly real-world interaction for improving robot policies. However, existing world-model-based methods largely treat world models as forward-only surrogates of the physical environment, leaving untapped a capability no physical environment affords: rewinding to past decision points to explore counterfactual futures. We introduce Rollback, a policy improvement paradigm that turns world models from passive simulators into counterfactual engines: it rolls back to critical decision points, branches into alternative actions, and contrasts their outcomes to attribute success to individual decisions. This concentrates imagination where it matters most and distills counterfactual outcomes into fine-grained, action-level supervision without a separately learned value model. We build a scalable training system that unifies distributed world-model search with asynchronous real-robot execution, forming a self-reinforcing cycle: real-world experience continually sharpens the world model, and a sharper world model in turn drives further policy improvement. We term the world-model side of this cycle test-time physics grounding, a capability that hand-built physics simulators lack: while the world model serves as the training environment, real transitions from the policy it trains correct its dynamics through a lightweight LoRA pipeline. Experiments on three challenging real-world tasks show that Rollback improves success rates over the strongest baseline by an absolute 10.67%, 11.11%, and 11.85% on block extraction, ball tossing, and tabletop curling, respectively, while achieving more stable improvement throughout training. Although initialized from teleoperated demonstrations far below human level, Rollback surpasses the average success rate of human players on both ball tossing and tabletop curling within 8 hours of training.
BibTeX
@misc{luo2026rollback,
  author = {Jiayi Luo and Yang Zhang and Liliang Chen and Nan Wang and Chen Gao and Boxiang Qiu and Zhuoxuan Li and Baorui Peng and Chuankang Li and Yuxiang Yan and Si Liu and Qingyun Sun and Shuicheng Yan and Maoqing Yao and Jianxin Li and Guanghui Ren},
  title = {{Rollback: Experience Backtracking in World Models for Policy Improvement}},
  year = {2026},
  eprint = {2610.36068},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36068},
}
Figure