PREFAIL: Identifying Precursors to Failures in Robotic Lift-and-Place Tasks to Improve Task Execution Performance

Zeyu Shangguan, Rajas Chitale, Rutvik Patel, Satyandra K. Gupta, Daniel SeitaarXiv 2026 · arXiv

TL;DR

PREFAIL is a policy-agnostic failure prediction framework designed for high-speed, non-prehensile robotic manipulation tasks like lift-and-place. By tracking the relative motion between an object and its carrier, it accurately forecasts drops before they happen. Crucially, it introduces a "historical backtracking" method to identify the exact "point of no return" ($t_{stop}$), allowing robots to dynamically modulate their speed to prevent failures without being overly conservative.

What Problem Does This Paper Address?

In industrial automation, non-prehensile manipulation—moving objects on flat trays or carriers without grasping them—is highly efficient but risky. When a robot arm moves at maximum speed, rapid direction changes generate large inertial forces. If these forces overcome friction, the object slips and falls, causing damage and costly downtime.

While running robots slowly prevents these failures, it severely hurts cycle times. Ideally, a robot should run at top speed and only slow down when a failure is imminent. However, existing failure-prediction methods are either reactive (detecting the failure after it has already occurred) or rely on out-of-distribution (OOD) detection, which frequently triggers false alarms during aggressive, high-speed maneuvers. Furthermore, current benchmarks fail to identify when an intervention is actually useful: if a failure is predicted too late, the robot's inertia makes a crash inevitable anyway.

Fig. 1: Our lift-and-place task involves a carrier (the pink plate in the image) mounted on a robot arm that handles an object (the cyan box) at the maximum allowable speed. Our method utilizes the visual observation and robot state values during a real-time robotic lift-and-place task, and predicts a continuous risk value (blue solid line) that represents the likelihood of failure. Once the risk is approaching the threshold (medium risk zone), the system must adjust and slow its speed, otherwise, the failure is inevitable after tstop (high risk). Since we will use sliding window during inference, there will be a warm-up period for collecting sufficient history information.

Methods

The authors propose PREFAIL, a framework that treats proactive failure prediction as an end-to-end time-series problem, outputting a continuous risk value between 0 (safe) and 1 (failed).

  1. Historical Backtracking (Data Labeling): To train the model, the authors developed a counterfactual labeling pipeline in simulation. Using a binary search strategy, they repeatedly replay a failing trajectory to find $t_{stop}$—the absolute last timestep at which an emergency stop can successfully prevent the object from falling. Timesteps before $t_{stop}$ are labeled low-risk, while timesteps after are high-risk. The continuous ground-truth risk curve is then smoothly interpolated using Hermite splines.
  2. Two-Branch Network Architecture:
  3. Observation Branch: Extracts visual features from multi-view cameras and computes the relative rotation and translation between the object and the carrier using 2D Iterative Closest Point (ICP) tracking on segmented contours.
  4. State Branch: Encodes proprioceptive robot signals (end-effector position, joint angles, and joint speeds) along with past risk estimates.
  5. Joint Loss Function: The network is optimized using a weighted combination of classification loss (to sharply distinguish the safety boundary at $t_{stop}$) and regression loss (to capture the continuous temporal evolution of risk).
Fig. 2: Overview of PREFAIL (see Sec. IV for details). For each sample with m time steps, the observation branch processes multi-view images to extract visual features P1, then uses the segmented contours of the target object and the carrier to calculate their relative motion to get the motion features P2. The state branch uses E3 to encode robot states and past risk values to produce features P3. We fuse and process features with E4 to output a continuous risk value Ri indicating the likelihood of a failure at each time step ti.

Datasets

The framework was validated on both simulated and physical setups:

  • Simulated Dataset (MuJoCo): Features 5,775 total trajectories. This includes a subset of 539 box-shaped object trajectories and a larger subset of 5,236 trajectories featuring highly diverse daily objects (e.g., bananas, toy pianos) to test geometric generalization.
  • Real-World Dataset: Consists of 254 trajectories collected using a physical UR5 robot arm transporting 5 types of box-shaped objects, including challenging packaged goods with shifting internal contents.

Key Insights and Ideas

  • Relative Motion is a Universal Signal: Instead of trying to analytically model complex, non-linear contact dynamics (like friction coefficients and surface geometry) online, PREFAIL relies on tracking subtle relative slip and rotation. This makes the method highly generalizable across different object shapes.
  • The Actionable Boundary: A failure prediction is only valuable if it leaves enough time for physical intervention. By explicitly labeling $t_{stop}$, the authors ensure that the model evaluates actionable timeliness rather than just raw predictive accuracy.
  • Smooth Supervision: Training the model on a continuous risk curve generated by splines provides a much stronger supervisory signal than discrete 0-or-1 step functions, helping the network distinguish early-stage stability from late-stage critical instability.
The historical backtracking pipeline uses binary search to find the exact $t_{stop}$ threshold.

Key Results

  • Dramatically Fewer False Alarms: Compared to competitive OOD baselines (such as Fail-Detect), which suffer from low precision because they default to treating almost every rapid trajectory as a failure, PREFAIL achieved 99.5% precision and 99.7% accuracy in simulation.
  • Optimized Intervention Timeliness: PREFAIL achieved an intervention timeliness ratio ($\eta$) of 0.93 to 0.96. This means it triggers an alert right before the critical $t_{stop}$ deadline, preventing premature, overly conservative stops.
  • Real-World Adaptive Speed Control: When deployed live on a physical UR5 robot, PREFAIL's continuous risk output was used to dynamically scale joint velocities. When risk increased, the robot smoothly slowed down to stabilize the object and then accelerated back to full speed once safe. This adaptive strategy successfully saved failing trajectories while incurring only a minor 0.1-second increase in overall cycle time.

Limitations and Future Work

  • Single-Object Limitation: The current framework only monitors a single object on a carrier. Multi-object scenarios would require additional reasoning to handle complex inter-object interactions.
  • Passive Interventions: PREFAIL currently only triggers speed reductions or stops; it does not plan active recovery maneuvers (like tilting the tray to catch a slipping object).
  • 2D ICP Vulnerabilities: The relative motion tracking relies on 2D projected contours, which can degrade under poor lighting or heavy occlusion. Incorporating depth sensing and 3D ICP could improve robustness.
  • Real-World Data Collection: While efficient in simulation, finding $t_{stop}$ via historical backtracking in the real world is highly labor-intensive due to the need for repeated physical resets.

Summary updated

Abstract

Non-prehensile manipulation enables flexible material handling with part carriers, but friction-based support makes high-speed motions failure-prone, while slower operation increases cycle time. Proactive failure prediction is therefore essential for efficient and reliable performance, yet existing approaches remain limited by key constraints, including sensitivity to dynamic actions and high dependence on known policy structures. Furthermore, existing methods and datasets lack a precise characterization of the latest intervention time, leaving it unclear whether a detected failure can still be prevented through timely intervention. In this paper, we investigate lift-and-place tasks for non-prehensile material handling manipulation and propose a more effective approach to predicting precursors to failures (PREFAIL) by analyzing the relative motion of target objects with respect to the carrier. We further introduce a dataset that precisely identifies the latest intervention time for risky manipulations, enabling rigorous evaluation of whether a failure prediction is actionable. We validate our approach on both simulation and real-world datasets. Our experimental results demonstrate that PREFAIL substantially improves both the accuracy and timeliness of responses to failure precursors.
BibTeX
@misc{shangguan2026prefail,
  author = {Zeyu Shangguan and Rajas Chitale and Rutvik Patel and Satyandra K. Gupta and Daniel Seita},
  title = {{PREFAIL: Identifying Precursors to Failures in Robotic Lift-and-Place Tasks to Improve Task Execution Performance}},
  year = {2026},
  eprint = {2607.16921},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.16921},
}
Figure