FailureGen: Scalable Robot Failure Data Generation with World Models

Akul Santhosh, Chaitra Sathe, Aastha JhunjhunwalaarXiv 2026 · arXiv

TL;DR

FailureGen is a novel framework that scales robot failure datasets by "splicing" recorded failure motions into successful task videos using generative world models. By transferring real-world error trajectories into new contexts, researchers can generate diverse, annotated synthetic failure data, significantly improving the performance of downstream failure detection models without the need for risky physical re-collection.

What Problem Does This Paper Address?

In the world of robotics, "success" is easy to record, but "failure" is a rare and expensive commodity. To build robust robots, we need models that can detect when things go wrong—like a gripper dropping a part or a robot missing its target. However, collecting these "long-tail" failure events at scale is difficult: repeating failures in the real world can damage hardware, and many errors depend on highly specific, hard-to-replicate conditions.

Existing solutions often rely on simulators, which can suffer from a "reality gap," or manual data augmentation, which lacks the complexity of real physical interactions. FailureGen addresses this by using action-conditioned world models to synthesize new failure scenarios based on real-world data.

Methods

The FailureGen pipeline consists of four main stages:

  1. World-Model Adaptation: The authors fine-tune a pretrained Cosmos world model (a generative model for physical AI) on a mix of successful and failed real-world robot executions. They use both forward-dynamics (predicting video from actions) and inverse-dynamics (predicting actions from video) to ensure the model understands how robot motion affects the scene.
  2. Source Annotation: An agentic pipeline using vision-capable Large Language Models (LLMs) identifies "donor" segments (the specific motion where a failure occurred) and "host" segments (successful recordings that provide the visual context).
  3. Motion Transfer: The "donor" failure motion is spliced into the "host" scene. The system can also apply spatial offsets, slightly shifting where the motion occurs to create even more variety.
  4. Generation & Filtering: The adapted Cosmos model generates a video rollout based on the host's initial state and the transferred motion. Finally, an automated pipeline filters out generations with visual artifacts or physically implausible results.
Figure 1: FailureGen architecture. Successful and failed source executions support world-model adaptation through forward- and inverse-dynamics supervision. At generation time, a successful host supplies visual context and nominal motion, while a failure donor supplies motion for a selected intervention. Motion transfer and optional spatial offsets construct the conditioning sequence for the adapted world model. Generated videos are labeled according to their observed events, with a cleanup filter for unreliable examples. Each retained example includes the video, its associated motion sequence, and temporal event labels.

Datasets

The researchers evaluated FailureGen on two distinct setups:

  • Industrial Bimanual Manipulation: An in-house dataset featuring a YAM robot performing complex material-handling tasks (picking and placing SMT reels). It includes 80 real episodes with failures like pickup misses and handover drops.
  • ViFailback: A public benchmark for robot failure detection. The authors focused on three specific tasks: placing objects in a black box, vial/test-tube rack placement, and drawer/can manipulation.

Key Insights and Ideas

  • Reusing the "Donor" Motion: The core "aha moment" is that a failure trajectory (the specific way a robot's arm moved when it dropped something) is a valuable asset. By treating this motion as a "donor," you can see how that same mistake would play out in dozens of different "host" environments.
  • World Models as High-Fidelity Simulators: Rather than trying to program complex physics for every object, FailureGen leverages the generative power of world models to "imagine" the consequences of a motion, such as an object bouncing off a table or falling into a bin.
  • Contextual Variation: As seen in Figure 2, the same "drop" motion from a donor results in a different visual outcome when applied to a host with a different object and starting position.
Figure 2: Failure transfer in industrial manipulation. Top: a real donor execution in which a reel falls outside the bin. Middle: the real host recording, which supplies the initial scene and object appearance. Bottom: the FailureGen continuation, which preserves the host context while reproducing the donor’s drop behavior with different timing and object motion. Columns show elapsed clip time; red outlines highlight the falling or dropped object.

Key Results

The results demonstrate that synthetic data generated by FailureGen provides a massive boost to failure detection models:

  • Higher Precision: In the industrial task, a detector trained with FailureGen-augmented data achieved an Average Precision of 0.51, compared to just 0.41 for a model trained only on real data.
  • Better Recall: The augmented model detected 33 out of 36 failure events (91.67%), whereas the real-only model caught only 28.
  • Zero False Alarms: Remarkably, the augmented model raised zero false alarms on clean episodes, while the real-only baseline struggled with "clean alarms."
  • Generalization: On the ViFailback tasks, using FailureGen data improved accuracy and recall across multiple model sizes (Qwen3-VL-2B and 8B), in some cases reaching 100% accuracy on placement tasks.
Table 2: Failure detection on the three-task ViFailback subset. Values show real-only →FailureGen- augmented performance (%). Bold values indicate improvements.

Limitations and Future Work

While FailureGen is a powerful tool for data scaling, the authors note a few hurdles:

  • Visual Artifacts: Generative world models aren't perfect; about 10% of generated videos had to be filtered out due to glitches or unrealistic physics.
  • Physical Plausibility: While the model is fine-tuned on physics, it can still occasionally produce "optimistic" outcomes where a failure should have occurred but the model generates a success (or vice versa).
  • Future Directions: The authors suggest expanding this method to more diverse robotic embodiments and exploring how these synthetic failures can be used not just for detection, but for teaching robots how to recover from errors.

Summary updated

Abstract

Robot failures are difficult to collect at scale, limiting the range of failure scenarios represented in robot learning datasets. We present FailureGen, a method for generating synthetic videos with temporal event annotations by reusing action trajectories from recorded failures. The method comprises four stages: (1) fine-tuning an action-conditioned Cosmos world model on successful and failed real executions; (2) annotating source recordings to identify failure-associated donor segments and compatible successful hosts; (3) transferring donor motion into host contexts, with optional spatial offsets, to generate continuous video rollouts; and (4) annotating and filtering the generated samples according to observed events and visual quality. We study two implementations with different tasks and robot configurations, with visual failure detection serving as one downstream application. In a preliminary comparison, a detector trained on real and generated data achieves an average precision of 0.51, compared with 0.41 using real data alone, and detects 33 of 36 failure intervals, compared with 28 of 36. By reusing observed failures in additional contexts, FailureGen expands the data available for studying robot failures and developing downstream learning methods.
BibTeX
@misc{santhosh2026failuregen,
  author = {Akul Santhosh and Chaitra Sathe and Aastha Jhunjhunwala},
  title = {{FailureGen: Scalable Robot Failure Data Generation with World Models}},
  year = {2026},
  eprint = {2611.22367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22367},
}
Figure