Old Experience, New Lessons: Relabeling Autonomous Robot Rollouts in Interactive Imitation Learning

Zhaoting Li, Cosimo Della Santina, Jens KoberarXiv 2026 · arXiv

In the world of interactive imitation learning (IIL), a robot learns by doing, with a human teacher stepping in to provide corrections when things go south. This process creates two very different piles of data: a small, precious set of human corrections and a massive mountain of autonomous robot "rollouts"—sequences where the robot acted on its own without a human label.

Most current methods either ignore that mountain of autonomous data or try to imitate it directly. The problem? If the robot wasn't perfect to begin with, imitating its past self just reinforces bad habits. A new paper, "Old Experience, New Lessons," introduces a framework called SeRA (Set-valued Relabeling of Autonomous experience) to solve this by letting the robot "re-evaluate" its past actions using its own improved, later-stage wisdom.

TL;DR

SeRA improves robot learning by using later, more advanced policy checkpoints to relabel earlier autonomous experiences. By treating these relabels as flexible "desired action sets" rather than rigid targets, the system avoids reinforcing errors and significantly outperforms existing interactive learning methods in both simulation and real-world tasks.

What Problem Does This Paper Address?

Interactive imitation learning is great because it focuses on the robot's failures. However, it’s data-inefficient. Human time is expensive, so corrections are sparse. Meanwhile, the robot generates tons of unlabeled autonomous data.

If we simply use Behavioral Cloning (BC) on this autonomous data, we might be training the robot to repeat "okay-ish" but suboptimal actions. If we try to "relabel" that data using the robot's current policy, we risk confirmation bias: the robot predicts an action, assumes it's correct because it matches its own current belief, and reinforces its own potential errors. The challenge is: How can we use unlabeled autonomous experience to improve the policy without accidentally training the robot to be confidently wrong?

Methods

The authors propose SeRA, which consists of three clever components:

  1. Cross-Round Policy Relabeling: As the robot learns, it gets better. SeRA uses policy checkpoints from later rounds to look back at observations from earlier rounds and suggest what the "correct" action should have been.
  2. Consistency Filtering: To avoid confirmation bias, SeRA doesn't just trust one checkpoint. It queries multiple later checkpoints. If they all agree on an action (low variance), the relabel is accepted. If they disagree, that step is discarded.
  3. Set-Valued Supervision: This is the "secret sauce." Instead of forcing the robot to match a relabeled action exactly, SeRA defines a desired action set—a region in action space. The size of this region is scaled by the distance between the original (suboptimal) action and the new relabel. This assumes the optimal action is somewhere in that direction, allowing the model to learn from the "intent" of the relabel without being shackled to its exact (and potentially imperfect) value.
SeRA Framework Overview

Finally, because they use Diffusion Policies (which predict chunks of actions at once), they introduce a step-masked denoising loss. This allows the model to learn from a mix of human corrections and accepted relabels within a single action sequence, masking out any steps where the supervision isn't trustworthy.

Fig. 2: Overview of SeRA. Later policy checkpoints relabel missing teacher actions, with checkpoint agreement determining the acceptance mask. Set-valued supervision uses relabels up to the first disagreement for unlabeled chunks (top) and combines accepted relabels with initial teacher actions for partially labeled chunks (bottom).

Datasets

The researchers evaluated SeRA across several challenging environments:

  • Simulation: Five tasks including PushT (pushing a T-block), and four manipulation tasks from the robosuite and MimicGen benchmarks (PickCan, Square, TwoArmLift, ThreePiece).
  • Real-World: A tomato-picking task where a physical robot must pick two tomatoes and place them in a bowl, dealing with visual occlusions and complex reach-in motions.

Key Insights and Ideas

  • "Later is Wiser": The core "aha moment" is that a policy trained on more human feedback can effectively "teach" its younger self by relabeling old data.
  • Agreement $\neq$ Correctness: The authors realized that even if multiple checkpoints agree, they might all be wrong. By using set-valued targets (regions) instead of point targets (exact values), they create a safety margin that prevents the robot from over-committing to its own errors.
  • Partial Supervision is Better than None: By using a masked loss, the robot can still learn from the "good" parts of a rollout even if a human only corrected a small fraction of it.

Key Results

SeRA consistently outperformed all baselines, including standard Set-Supervised Diffusion Policy (SDP) and methods that imitate historical robot actions (like IWR or Sirius).

  • Simulation: In offline learning (training on a fixed dataset), SeRA achieved near-perfect success rates on tasks where other methods struggled. For example, on the "Square" task, SeRA hit 96.3% success, while the base SDP method was at 90.8%.
  • Real Robot: On the tomato-picking task, SeRA achieved an 80% success rate (16/20 trials). In comparison, the base SDP method only managed 40%, and methods that imitated historical robot actions performed even worse.

Limitations and Future Work

While powerful, SeRA has its bounds:

  • The Optimality Assumption: It assumes the "desired set" actually contains the optimal action. If the checkpoints are consistently wrong in the same way, the set might not help.
  • Fixed Radius: Currently, the "slack" given to the robot (the radius of the action set) is fixed. Future work could make this dynamic, perhaps using the variance between checkpoints to decide how much to trust a relabel.
  • Discarded Data: Currently, if checkpoints disagree, the data is just ignored. This "high disagreement" data could instead be used to signal where the robot is most confused, potentially guiding a human to provide more targeted corrections.

Paper Citation: Zhaoting Li, Cosimo Della Santina, and Jens Kober. "Old Experience, New Lessons: Relabeling Autonomous Robot Rollouts in Interactive Imitation Learning." arXiv:2611.24273v1, 2026.

Summary updated

Abstract

Interactive imitation learning enables robots to learn from human interventions during deployment. These interactions naturally produce sparse human corrections and abundant autonomous robot experience without human action labels. Yet existing methods either leave this experience unused for action supervision or imitate historical robot actions, which can reinforce suboptimal behavior. To address this limitation, we propose Set-valued Relabeling of Autonomous experience (SeRA), a framework that learns from autonomous robot experience through set-valued supervision of policy relabels. Later policy checkpoints relabel this experience, and agreement across checkpoints filters the relabels. Since agreement does not guarantee correctness, SeRA constructs desired action sets around accepted relabels. Following Set-Supervised Diffusion Policy (SDP), it trains an action-chunking diffusion policy to increase the probability of actions within these sets. A step-masked denoising loss combines accepted relabels with available human corrections in partially supervised chunks. Experiments across five simulated manipulation tasks in online and offline settings show that SeRA improves policy performance over SDP and outperforms baselines that imitate historical robot actions or exact relabels. On a real-robot tomato-picking task, SeRA achieves 80% success, compared with 40% for SDP, using the same human feedback and training budget. These results demonstrate that relabeled autonomous robot experience can complement human corrections to improve policy performance.
BibTeX
@misc{li2026old,
  author = {Zhaoting Li and Cosimo Della Santina and Jens Kober},
  title = {{Old Experience, New Lessons: Relabeling Autonomous Robot Rollouts in Interactive Imitation Learning}},
  year = {2026},
  eprint = {2611.24273},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24273},
}
Figure