Curating Demonstrations using Online Experience

Yu‐Jen Chen, Alec Lessing, Yuejiang Liu, Chelsea Finn2025 · DOI

TL;DR

Demo-SCORE (Curating Demonstrations using Online Experience) is a self-curation framework that allows robots to automatically filter out "bad" or unreliable demonstrations from large, messy datasets. By training a classifier on its own trial-and-error rollouts, the robot learns to identify which human strategies are actually reproducible, leading to a 15–35% boost in success rates without any manual data labeling.

What Problem Does This Paper Address?

Imitation learning (IL) relies on the quality of human demonstrations. However, large-scale datasets are often "heterogeneous"—meaning they contain a mix of different strategies, some of which might be suboptimal, rare, or simply too difficult for a robot's specific hardware to replicate reliably.

For example, a human might pick up a spoon by the very tip of the handle. While a human can do this easily, a robot's gripper might find this strategy unstable, leading to frequent drops. If a robot is trained on all these demonstrations indiscriminately, it may attempt these "unreliable" strategies at test time and fail. Manually sifting through thousands of videos to find these edge cases is prohibitively expensive and slow. The authors ask: Can a robot use its own experience to figure out which demonstrations it should actually follow?

Methods

The authors propose Demo-SCORE, a four-step pipeline for automated data curation:

  1. Train an Initial Policy: Train a base policy (using architectures like Diffusion Policy or ACT) on the entire, unfiltered dataset. The goal here is to capture the full distribution of behaviors, even the bad ones.
  2. Collect Online Rollouts: Run the initial policy in the environment at various stages of training (checkpoints). Each attempt (rollout) is automatically labeled as a "success" or "failure" based on a binary indicator (e.g., did the peg go in the hole?).
  3. Train a Data Quality Classifier: Train a classifier (a simple MLP) to distinguish between the successful and failed rollouts. Crucially, they use cross-validation across different training checkpoints to ensure the classifier doesn't just overfit to one specific version of the robot's behavior.
  4. Filter and Retrain: Use the trained classifier to "score" the original human demonstrations. Demonstrations that look like the robot's failed attempts are discarded. Finally, a new, "final" policy is trained on this curated, high-quality subset.

Datasets

The researchers evaluated Demo-SCORE across both simulated and real-world environments:

  • Simulation (Robosuite & ALOHA): Bimanual peg insertion and a square peg task. They intentionally created "lopsided" mixtures of scripted and human data to test the filter's robustness.
  • Real-World (ALOHA): Five diverse manipulation tasks:
  • Placing a spoon in a rack.
  • Transferring a chocolate bar between plates.
  • Picking up a Sharpie from a cluttered scene.
  • Picking up a strawberry.
  • The Jellybean Task: A complex, long-horizon task involving five sub-steps (retrieving a cup, filling it with jellybeans, etc.).

Key Insights and Ideas

  • The Robot is the Best Judge of Itself: Humans often can't tell which strategies will be hard for a robot. Demo-SCORE grounds data curation in the robot's actual embodiment and limitations.
  • Failure is Information: Most IL methods only look at successful demonstrations. Demo-SCORE uses the robot's failures during training to define what "bad" data looks like.
  • Cross-Validation is Critical: A classifier trained on rollouts from just one policy checkpoint tends to overfit. By using rollouts from multiple points in the training process, the classifier learns a more general representation of "reliability" that applies better to the original human data.
  • Granularity Matters: The method works whether you filter entire trajectories or just specific "chunks" of a long task, making it flexible for complex, multi-stage behaviors.

Key Results

  • Performance Boost: Policies trained with Demo-SCORE achieved 15–35% higher absolute success rates compared to policies trained on the full, unfiltered datasets.
  • Real-World Success: In real-world ALOHA tasks, Demo-SCORE improved success rates by nearly 30% on average. On the difficult Jellybean task, the base policy had a 0% full-task success rate, while the Demo-SCORE curated policy reached 20%.
  • Efficiency: The method is highly sample-efficient. Experiments showed that as few as 10 rollouts per checkpoint (50 total) were enough to train an effective classifier.
  • Comparison: Demo-SCORE consistently outperformed prior "Auto-IL" methods and loss-based weighting schemes, particularly in scenarios with highly varied human strategies.

Limitations and Future Work

  • Initial Policy Quality: Demo-SCORE assumes the initial policy is capable of at least attempting the strategies in the dataset. If the initial policy suffers from "mode collapse" (only learning one way to do things), the rollouts won't represent the full dataset, and the filter might fail.
  • Success Indicators: The method requires an automated way to detect success/failure during rollouts. While often available in sim, this can be harder to implement in the real world without specialized sensors or secondary vision models.
  • State Coverage: By filtering data, you naturally reduce the diversity of the training set. While this improves reliability for a specific task, it might theoretically reduce the robot's ability to recover from highly unusual out-of-distribution states.
  • Future Direction: The authors suggest exploring how this curation could be integrated into "pre-training" large-scale foundation models for robotics.

Summary updated

Abstract

Many robot demonstration datasets contain heterogeneous demonstrations of varying quality.This heterogeneity may benefit policy pre-training, but can hinder robot performance when used with a final imitation learning objective.In particular, some strategies in the data may be less reliable than others or may be underrepresented in the data, leading to poor performance when such strategies are sampled at test time.Moreover, such unreliable or underrepresented strategies can be difficult even for people to discern, and sifting through demonstration datasets is time-consuming and costly.On the other hand, policy performance when trained on such demonstrations can reflect the reliability of different strategies.We thus propose for robots to self-curate based on online robot experience (Demo-SCORE).More specifically, we train and cross-validate a classifier to discern successful policy roll-outs from unsuccessful ones and use the classifier to filter heterogeneous demonstration datasets.Our experiments in simulation and the real world show that Demo-SCORE can effectively identify suboptimal demonstrations without manual curation.Notably, Demo-SCORE achieves over 15-35% higher absolute success rate in the resulting policy compared to the base policy trained with all original demonstrations.
BibTeX
@misc{chen2025curating,
  author = {Yu‐Jen Chen and Alec Lessing and Yuejiang Liu and Chelsea Finn},
  title = {{Curating Demonstrations using Online Experience}},
  year = {2025},
  doi = {10.15607/rss.2025.xxi.071},
}
Figure