TL;DR
PREFAIL is a policy-agnostic failure prediction framework designed for high-speed, non-prehensile robotic manipulation tasks like lift-and-place. By tracking the relative motion between an object and its carrier, it accurately forecasts drops before they happen. Crucially, it introduces a "historical backtracking" method to identify the exact "point of no return" ($t_{stop}$), allowing robots to dynamically modulate their speed to prevent failures without being overly conservative.
What Problem Does This Paper Address?
In industrial automation, non-prehensile manipulation—moving objects on flat trays or carriers without grasping them—is highly efficient but risky. When a robot arm moves at maximum speed, rapid direction changes generate large inertial forces. If these forces overcome friction, the object slips and falls, causing damage and costly downtime.
While running robots slowly prevents these failures, it severely hurts cycle times. Ideally, a robot should run at top speed and only slow down when a failure is imminent. However, existing failure-prediction methods are either reactive (detecting the failure after it has already occurred) or rely on out-of-distribution (OOD) detection, which frequently triggers false alarms during aggressive, high-speed maneuvers. Furthermore, current benchmarks fail to identify when an intervention is actually useful: if a failure is predicted too late, the robot's inertia makes a crash inevitable anyway.
Methods
The authors propose PREFAIL, a framework that treats proactive failure prediction as an end-to-end time-series problem, outputting a continuous risk value between 0 (safe) and 1 (failed).
- Historical Backtracking (Data Labeling): To train the model, the authors developed a counterfactual labeling pipeline in simulation. Using a binary search strategy, they repeatedly replay a failing trajectory to find $t_{stop}$—the absolute last timestep at which an emergency stop can successfully prevent the object from falling. Timesteps before $t_{stop}$ are labeled low-risk, while timesteps after are high-risk. The continuous ground-truth risk curve is then smoothly interpolated using Hermite splines.
- Two-Branch Network Architecture:
- Observation Branch: Extracts visual features from multi-view cameras and computes the relative rotation and translation between the object and the carrier using 2D Iterative Closest Point (ICP) tracking on segmented contours.
- State Branch: Encodes proprioceptive robot signals (end-effector position, joint angles, and joint speeds) along with past risk estimates.
- Joint Loss Function: The network is optimized using a weighted combination of classification loss (to sharply distinguish the safety boundary at $t_{stop}$) and regression loss (to capture the continuous temporal evolution of risk).
Datasets
The framework was validated on both simulated and physical setups:
- Simulated Dataset (MuJoCo): Features 5,775 total trajectories. This includes a subset of 539 box-shaped object trajectories and a larger subset of 5,236 trajectories featuring highly diverse daily objects (e.g., bananas, toy pianos) to test geometric generalization.
- Real-World Dataset: Consists of 254 trajectories collected using a physical UR5 robot arm transporting 5 types of box-shaped objects, including challenging packaged goods with shifting internal contents.
Key Insights and Ideas
- Relative Motion is a Universal Signal: Instead of trying to analytically model complex, non-linear contact dynamics (like friction coefficients and surface geometry) online, PREFAIL relies on tracking subtle relative slip and rotation. This makes the method highly generalizable across different object shapes.
- The Actionable Boundary: A failure prediction is only valuable if it leaves enough time for physical intervention. By explicitly labeling $t_{stop}$, the authors ensure that the model evaluates actionable timeliness rather than just raw predictive accuracy.
- Smooth Supervision: Training the model on a continuous risk curve generated by splines provides a much stronger supervisory signal than discrete 0-or-1 step functions, helping the network distinguish early-stage stability from late-stage critical instability.
Key Results
- Dramatically Fewer False Alarms: Compared to competitive OOD baselines (such as Fail-Detect), which suffer from low precision because they default to treating almost every rapid trajectory as a failure, PREFAIL achieved 99.5% precision and 99.7% accuracy in simulation.
- Optimized Intervention Timeliness: PREFAIL achieved an intervention timeliness ratio ($\eta$) of 0.93 to 0.96. This means it triggers an alert right before the critical $t_{stop}$ deadline, preventing premature, overly conservative stops.
- Real-World Adaptive Speed Control: When deployed live on a physical UR5 robot, PREFAIL's continuous risk output was used to dynamically scale joint velocities. When risk increased, the robot smoothly slowed down to stabilize the object and then accelerated back to full speed once safe. This adaptive strategy successfully saved failing trajectories while incurring only a minor 0.1-second increase in overall cycle time.
Limitations and Future Work
- Single-Object Limitation: The current framework only monitors a single object on a carrier. Multi-object scenarios would require additional reasoning to handle complex inter-object interactions.
- Passive Interventions: PREFAIL currently only triggers speed reductions or stops; it does not plan active recovery maneuvers (like tilting the tray to catch a slipping object).
- 2D ICP Vulnerabilities: The relative motion tracking relies on 2D projected contours, which can degrade under poor lighting or heavy occlusion. Incorporating depth sensing and 3D ICP could improve robustness.
- Real-World Data Collection: While efficient in simulation, finding $t_{stop}$ via historical backtracking in the real world is highly labor-intensive due to the need for repeated physical resets.
Summary updated