Rollout then Recycle: 3,718 Labelled Real-Robot Episodes from Benchmarking 7 Policies on 12 Tasks

Praveen Selvaraj, Lorenzo Uttini, Ville KuosmanenarXiv 2026 · arXiv

TL;DR

Researchers have released ArmnetBench v0.1, a unique dataset of 3,718 real-robot episodes that "recycles" the failures typically discarded after benchmarking. Unlike most datasets that focus on perfect demonstrations, 73% of this corpus consists of labeled failures from seven different state-of-the-art policies across 12 tabletop tasks, providing a goldmine for training reward models and success classifiers.

What Problem Does This Paper Address?

In the world of robotic learning, we are currently "success-obsessed." Large-scale datasets like Open X-Embodiment and DROID consist almost entirely of successful demonstrations. While these are great for imitation learning, they leave a massive gap: robots don't know what failure looks like.

When researchers benchmark new policies, they inevitably generate thousands of rollout episodes, many of which are failures. Usually, these are used to calculate a success rate and then deleted. The authors argue that these "failed" rollouts are actually highly valuable data points for teaching robots to recognize their own mistakes, building robust reward models, and improving world models by showing how unsuccessful interactions evolve.

Methods

The authors collected data using a standardized "arm farm" consisting of three co-located SO-101 robot cells (two single-arm and one bimanual).

To generate the data, they took 12 distinct tabletop tasks and provided 50 teleoperated demonstrations for each. They then trained or fine-tuned seven different policy architectures on these same demonstrations:

  • Imitation Learning: ACT and Diffusion Policy.
  • Vision-Language-Action (VLA): SmolVLA, $\pi_0$, $\pi_{0.5}$, GR00T N1.7, and MolmoAct 2.

By running these policies on the same hardware and tasks, they created a controlled environment where the primary variable is the policy itself. Each rollout was recorded with three camera views at 20 fps and manually labeled by an operator as successful, suboptimal, or failure.

Figure 5: The 12 tabletop tasks used in the benchmark, ranging from simple block stacking to complex bimanual towel folding.

Datasets

The released dataset contains 3,718 episodes (~26 hours of footage) and is available on the Hugging Face Hub in LeRobot v3.0 format. It includes:

  • 600 Teleoperated Demonstrations: The "gold standard" successes used for training.
  • 3,118 Policy Rollouts: The core of the dataset, featuring 724 successes, 106 suboptimal attempts, and 2,288 failures.

The data is split across two main repositories: one for single-arm tasks (2,499 episodes) and one for bimanual tasks (1,219 episodes).

Key Insights and Ideas

  • The Value of Negative Data: By releasing failures, the authors enable the community to train "success classifiers." If a robot can recognize it has failed, it can trigger a recovery behavior or request human help.
  • Shared Setup Comparison: Because all seven policies were tested on the exact same tasks and hardware, researchers can study how different architectures (e.g., a transformer-based ACT vs. a diffusion-based policy) fail differently on the same physical challenges.
  • Label Leakage Warning: The authors highlight a subtle but critical issue: because successful episodes were stopped immediately upon completion while failures ran longer, a model might "cheat" by using episode length to predict success. They advise researchers to use length-matched or clipped inputs to avoid this bias.

Key Results

The benchmark revealed a high diversity in performance across policies and tasks:

  • Overall Failure Rate: A staggering 73.4% of policy rollouts were failures, emphasizing how difficult these "simple" tabletop tasks remain for current SOTA models.
  • Policy Performance: $\pi_{0.5}$ showed the highest success count (171), while SmolVLA struggled the most (63 successes).
  • Task Difficulty: Some tasks proved nearly impossible for the tested policies; for instance, the cable_clip task resulted in zero successes across 270 rollouts, despite having 50 successful demonstrations to learn from.

Limitations and Future Work

The authors are transparent about several limitations:

  • Subjective Labeling: Labels rely on a single operator's judgment.
  • Binary Nature: Only 3.4% of rollouts were labeled "suboptimal," meaning the dataset lacks a fine-grained "partial progress" signal.
  • Environmental Drift: Over the course of the evaluation, some physical objects wore down, and two cameras suffered from minor misalignment or blurriness, which might allow a model to identify a policy based on visual artifacts rather than behavior.
  • Reproducibility: Due to metadata limitations, the exact leaderboard from the original benchmark cannot be perfectly reproduced from this specific release, though the raw data for all episodes is present.

This work serves as a call to action for the robotics community: don't delete your failures—recycle them.

Summary updated

Abstract

We release all 3,718 episodes from the ArmnetBench v0.1 evaluation (arXiv:2607.24481), which contains 600 teleoperated demonstrations and 3,118 rollouts of seven imitation-learning and vision-language-action policies on 12 tabletop tasks, recorded on low-cost SO-101 arms; 73% of the rollouts are failures. Every policy for a task was trained or fine-tuned on the same 50 demonstrations and rolled out on the same cell, so the corpus contains many policies' failures on shared tasks. An operator labelled every rollout successful, suboptimal or failure. Each episode has three camera views, joint states and actions and is available on the Hugging Face Hub in LeRobot v3.0 and a reward-model format. This note documents how the episodes were collected and the limitations that could affect their use.
BibTeX
@misc{selvaraj2026rollout,
  author = {Praveen Selvaraj and Lorenzo Uttini and Ville Kuosmanen},
  title = {{Rollout then Recycle: 3,718 Labelled Real-Robot Episodes from Benchmarking 7 Policies on 12 Tasks}},
  year = {2026},
  eprint = {2611.03168},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03168},
}
Figure