In the world of interactive imitation learning (IIL), a robot learns by doing, with a human teacher stepping in to provide corrections when things go south. This process creates two very different piles of data: a small, precious set of human corrections and a massive mountain of autonomous robot "rollouts"—sequences where the robot acted on its own without a human label.
Most current methods either ignore that mountain of autonomous data or try to imitate it directly. The problem? If the robot wasn't perfect to begin with, imitating its past self just reinforces bad habits. A new paper, "Old Experience, New Lessons," introduces a framework called SeRA (Set-valued Relabeling of Autonomous experience) to solve this by letting the robot "re-evaluate" its past actions using its own improved, later-stage wisdom.
TL;DR
SeRA improves robot learning by using later, more advanced policy checkpoints to relabel earlier autonomous experiences. By treating these relabels as flexible "desired action sets" rather than rigid targets, the system avoids reinforcing errors and significantly outperforms existing interactive learning methods in both simulation and real-world tasks.
What Problem Does This Paper Address?
Interactive imitation learning is great because it focuses on the robot's failures. However, it’s data-inefficient. Human time is expensive, so corrections are sparse. Meanwhile, the robot generates tons of unlabeled autonomous data.
If we simply use Behavioral Cloning (BC) on this autonomous data, we might be training the robot to repeat "okay-ish" but suboptimal actions. If we try to "relabel" that data using the robot's current policy, we risk confirmation bias: the robot predicts an action, assumes it's correct because it matches its own current belief, and reinforces its own potential errors. The challenge is: How can we use unlabeled autonomous experience to improve the policy without accidentally training the robot to be confidently wrong?
Methods
The authors propose SeRA, which consists of three clever components:
- Cross-Round Policy Relabeling: As the robot learns, it gets better. SeRA uses policy checkpoints from later rounds to look back at observations from earlier rounds and suggest what the "correct" action should have been.
- Consistency Filtering: To avoid confirmation bias, SeRA doesn't just trust one checkpoint. It queries multiple later checkpoints. If they all agree on an action (low variance), the relabel is accepted. If they disagree, that step is discarded.
- Set-Valued Supervision: This is the "secret sauce." Instead of forcing the robot to match a relabeled action exactly, SeRA defines a desired action set—a region in action space. The size of this region is scaled by the distance between the original (suboptimal) action and the new relabel. This assumes the optimal action is somewhere in that direction, allowing the model to learn from the "intent" of the relabel without being shackled to its exact (and potentially imperfect) value.
Finally, because they use Diffusion Policies (which predict chunks of actions at once), they introduce a step-masked denoising loss. This allows the model to learn from a mix of human corrections and accepted relabels within a single action sequence, masking out any steps where the supervision isn't trustworthy.
Datasets
The researchers evaluated SeRA across several challenging environments:
- Simulation: Five tasks including PushT (pushing a T-block), and four manipulation tasks from the robosuite and MimicGen benchmarks (PickCan, Square, TwoArmLift, ThreePiece).
- Real-World: A tomato-picking task where a physical robot must pick two tomatoes and place them in a bowl, dealing with visual occlusions and complex reach-in motions.
Key Insights and Ideas
- "Later is Wiser": The core "aha moment" is that a policy trained on more human feedback can effectively "teach" its younger self by relabeling old data.
- Agreement $\neq$ Correctness: The authors realized that even if multiple checkpoints agree, they might all be wrong. By using set-valued targets (regions) instead of point targets (exact values), they create a safety margin that prevents the robot from over-committing to its own errors.
- Partial Supervision is Better than None: By using a masked loss, the robot can still learn from the "good" parts of a rollout even if a human only corrected a small fraction of it.
Key Results
SeRA consistently outperformed all baselines, including standard Set-Supervised Diffusion Policy (SDP) and methods that imitate historical robot actions (like IWR or Sirius).
- Simulation: In offline learning (training on a fixed dataset), SeRA achieved near-perfect success rates on tasks where other methods struggled. For example, on the "Square" task, SeRA hit 96.3% success, while the base SDP method was at 90.8%.
- Real Robot: On the tomato-picking task, SeRA achieved an 80% success rate (16/20 trials). In comparison, the base SDP method only managed 40%, and methods that imitated historical robot actions performed even worse.
Limitations and Future Work
While powerful, SeRA has its bounds:
- The Optimality Assumption: It assumes the "desired set" actually contains the optimal action. If the checkpoints are consistently wrong in the same way, the set might not help.
- Fixed Radius: Currently, the "slack" given to the robot (the radius of the action set) is fixed. Future work could make this dynamic, perhaps using the variance between checkpoints to decide how much to trust a relabel.
- Discarded Data: Currently, if checkpoints disagree, the data is just ignored. This "high disagreement" data could instead be used to signal where the robot is most confused, potentially guiding a human to provide more targeted corrections.
Paper Citation: Zhaoting Li, Cosimo Della Santina, and Jens Kober. "Old Experience, New Lessons: Relabeling Autonomous Robot Rollouts in Interactive Imitation Learning." arXiv:2611.24273v1, 2026.
Summary updated