TL;DR
FailureGen is a novel framework that scales robot failure datasets by "splicing" recorded failure motions into successful task videos using generative world models. By transferring real-world error trajectories into new contexts, researchers can generate diverse, annotated synthetic failure data, significantly improving the performance of downstream failure detection models without the need for risky physical re-collection.
What Problem Does This Paper Address?
In the world of robotics, "success" is easy to record, but "failure" is a rare and expensive commodity. To build robust robots, we need models that can detect when things go wrong—like a gripper dropping a part or a robot missing its target. However, collecting these "long-tail" failure events at scale is difficult: repeating failures in the real world can damage hardware, and many errors depend on highly specific, hard-to-replicate conditions.
Existing solutions often rely on simulators, which can suffer from a "reality gap," or manual data augmentation, which lacks the complexity of real physical interactions. FailureGen addresses this by using action-conditioned world models to synthesize new failure scenarios based on real-world data.
Methods
The FailureGen pipeline consists of four main stages:
- World-Model Adaptation: The authors fine-tune a pretrained Cosmos world model (a generative model for physical AI) on a mix of successful and failed real-world robot executions. They use both forward-dynamics (predicting video from actions) and inverse-dynamics (predicting actions from video) to ensure the model understands how robot motion affects the scene.
- Source Annotation: An agentic pipeline using vision-capable Large Language Models (LLMs) identifies "donor" segments (the specific motion where a failure occurred) and "host" segments (successful recordings that provide the visual context).
- Motion Transfer: The "donor" failure motion is spliced into the "host" scene. The system can also apply spatial offsets, slightly shifting where the motion occurs to create even more variety.
- Generation & Filtering: The adapted Cosmos model generates a video rollout based on the host's initial state and the transferred motion. Finally, an automated pipeline filters out generations with visual artifacts or physically implausible results.
Datasets
The researchers evaluated FailureGen on two distinct setups:
- Industrial Bimanual Manipulation: An in-house dataset featuring a YAM robot performing complex material-handling tasks (picking and placing SMT reels). It includes 80 real episodes with failures like pickup misses and handover drops.
- ViFailback: A public benchmark for robot failure detection. The authors focused on three specific tasks: placing objects in a black box, vial/test-tube rack placement, and drawer/can manipulation.
Key Insights and Ideas
- Reusing the "Donor" Motion: The core "aha moment" is that a failure trajectory (the specific way a robot's arm moved when it dropped something) is a valuable asset. By treating this motion as a "donor," you can see how that same mistake would play out in dozens of different "host" environments.
- World Models as High-Fidelity Simulators: Rather than trying to program complex physics for every object, FailureGen leverages the generative power of world models to "imagine" the consequences of a motion, such as an object bouncing off a table or falling into a bin.
- Contextual Variation: As seen in Figure 2, the same "drop" motion from a donor results in a different visual outcome when applied to a host with a different object and starting position.
Key Results
The results demonstrate that synthetic data generated by FailureGen provides a massive boost to failure detection models:
- Higher Precision: In the industrial task, a detector trained with FailureGen-augmented data achieved an Average Precision of 0.51, compared to just 0.41 for a model trained only on real data.
- Better Recall: The augmented model detected 33 out of 36 failure events (91.67%), whereas the real-only model caught only 28.
- Zero False Alarms: Remarkably, the augmented model raised zero false alarms on clean episodes, while the real-only baseline struggled with "clean alarms."
- Generalization: On the ViFailback tasks, using FailureGen data improved accuracy and recall across multiple model sizes (Qwen3-VL-2B and 8B), in some cases reaching 100% accuracy on placement tasks.
Limitations and Future Work
While FailureGen is a powerful tool for data scaling, the authors note a few hurdles:
- Visual Artifacts: Generative world models aren't perfect; about 10% of generated videos had to be filtered out due to glitches or unrealistic physics.
- Physical Plausibility: While the model is fine-tuned on physics, it can still occasionally produce "optimistic" outcomes where a failure should have occurred but the model generates a success (or vice versa).
- Future Directions: The authors suggest expanding this method to more diverse robotic embodiments and exploring how these synthetic failures can be used not just for detection, but for teaching robots how to recover from errors.
Summary updated