TL;DR
Researchers have released ArmnetBench v0.1, a unique dataset of 3,718 real-robot episodes that "recycles" the failures typically discarded after benchmarking. Unlike most datasets that focus on perfect demonstrations, 73% of this corpus consists of labeled failures from seven different state-of-the-art policies across 12 tabletop tasks, providing a goldmine for training reward models and success classifiers.
What Problem Does This Paper Address?
In the world of robotic learning, we are currently "success-obsessed." Large-scale datasets like Open X-Embodiment and DROID consist almost entirely of successful demonstrations. While these are great for imitation learning, they leave a massive gap: robots don't know what failure looks like.
When researchers benchmark new policies, they inevitably generate thousands of rollout episodes, many of which are failures. Usually, these are used to calculate a success rate and then deleted. The authors argue that these "failed" rollouts are actually highly valuable data points for teaching robots to recognize their own mistakes, building robust reward models, and improving world models by showing how unsuccessful interactions evolve.
Methods
The authors collected data using a standardized "arm farm" consisting of three co-located SO-101 robot cells (two single-arm and one bimanual).
To generate the data, they took 12 distinct tabletop tasks and provided 50 teleoperated demonstrations for each. They then trained or fine-tuned seven different policy architectures on these same demonstrations:
- Imitation Learning: ACT and Diffusion Policy.
- Vision-Language-Action (VLA): SmolVLA, $\pi_0$, $\pi_{0.5}$, GR00T N1.7, and MolmoAct 2.
By running these policies on the same hardware and tasks, they created a controlled environment where the primary variable is the policy itself. Each rollout was recorded with three camera views at 20 fps and manually labeled by an operator as successful, suboptimal, or failure.
Datasets
The released dataset contains 3,718 episodes (~26 hours of footage) and is available on the Hugging Face Hub in LeRobot v3.0 format. It includes:
- 600 Teleoperated Demonstrations: The "gold standard" successes used for training.
- 3,118 Policy Rollouts: The core of the dataset, featuring 724 successes, 106 suboptimal attempts, and 2,288 failures.
The data is split across two main repositories: one for single-arm tasks (2,499 episodes) and one for bimanual tasks (1,219 episodes).
Key Insights and Ideas
- The Value of Negative Data: By releasing failures, the authors enable the community to train "success classifiers." If a robot can recognize it has failed, it can trigger a recovery behavior or request human help.
- Shared Setup Comparison: Because all seven policies were tested on the exact same tasks and hardware, researchers can study how different architectures (e.g., a transformer-based ACT vs. a diffusion-based policy) fail differently on the same physical challenges.
- Label Leakage Warning: The authors highlight a subtle but critical issue: because successful episodes were stopped immediately upon completion while failures ran longer, a model might "cheat" by using episode length to predict success. They advise researchers to use length-matched or clipped inputs to avoid this bias.
Key Results
The benchmark revealed a high diversity in performance across policies and tasks:
- Overall Failure Rate: A staggering 73.4% of policy rollouts were failures, emphasizing how difficult these "simple" tabletop tasks remain for current SOTA models.
- Policy Performance: $\pi_{0.5}$ showed the highest success count (171), while SmolVLA struggled the most (63 successes).
- Task Difficulty: Some tasks proved nearly impossible for the tested policies; for instance, the
cable_cliptask resulted in zero successes across 270 rollouts, despite having 50 successful demonstrations to learn from.
Limitations and Future Work
The authors are transparent about several limitations:
- Subjective Labeling: Labels rely on a single operator's judgment.
- Binary Nature: Only 3.4% of rollouts were labeled "suboptimal," meaning the dataset lacks a fine-grained "partial progress" signal.
- Environmental Drift: Over the course of the evaluation, some physical objects wore down, and two cameras suffered from minor misalignment or blurriness, which might allow a model to identify a policy based on visual artifacts rather than behavior.
- Reproducibility: Due to metadata limitations, the exact leaderboard from the original benchmark cannot be perfectly reproduced from this specific release, though the raw data for all episodes is present.
This work serves as a call to action for the robotics community: don't delete your failures—recycle them.
Summary updated