TL;DR
GTP-FA (Grasp-Then-Plan with Failure Attribution) is a closed-loop framework that decouples robotic manipulation into distinct grasping and planning stages while intelligently diagnosing execution failures. By accurately attributing whether a failure stems from a faulty grasp or a downstream planning error, the system routes optimization pressure to the correct module, significantly boosting the success rates of reinforcement learning, imitation learning, and vision-language-action (VLA) models.
What Problem Does This Paper Address?
In long-horizon robotic manipulation, a task's success is heavily dictated by the initial grasp. For instance, if a robot picks up a tool by its functional end rather than its handle, even the most advanced motion planning policy will inevitably fail later in the execution.
This creates a challenging "credit assignment" problem due to the tight coupling between grasping and planning. When a failure occurs late in a trajectory, standard end-to-end learning algorithms often misattribute the error to the downstream planning policy rather than the fundamentally flawed initial grasp. This misattribution contaminates training data and severely weakens the robot's ability to generalize to new scenarios. While separating the pipeline into "grasp-then-plan" stages helps, traditional methods still fail to communicate downstream execution realities back to the grasp generator.
Methods
To close this loop, the authors introduce GTP-FA, which operates as an iterative execute-diagnose-update pipeline.
The framework relies on three core components:
- Failure Attribution Discriminator ($D$): A lightweight model that categorizes execution failures into three distinct modes: task-irrelevant grasp contact (FM-G1), unstable grasping/slippage (FM-G2), or downstream planning insufficiency (FM-P). Training labels are generated automatically via weak supervision from repeated trials under fixed grasp conditions.
- Grasp-Conditioned Diagnostic Representation Space ($E$): To ensure the discriminator generalizes to entirely unseen test-time grasps, this component maps post-grasp initialization states into an embedding space using a supervised contrastive objective. It enforces a local-consistency prior: grasps that result in similar physical states should share similar failure distributions.
- Diagnosis-Driven Bidirectional Optimization: The system calculates responsibility weights based on the fused outputs of $D$ and $E$.
- If the failure is grasp-dominant, the grasp selection module is updated by injecting task-level semantic priors (via a Vision-Language Model) and diagnostic risk penalties to suppress unstable candidate grasps.
- If the failure is planning-dominant, the downstream policy is refined using "hard-planning" sample mining, training distribution reshaping, and responsibility-weighted loss functions.
Datasets
The authors evaluated GTP-FA across both simulated and physical environments:
- Simulation: Eight long-horizon manipulation tasks from the ManiSkill3 benchmark, including StackCube, PokeCube, LiftPegUpright, PlaceSphere, PickCube, PushCube, PullCube, and PullCubeTool.
- Real-Robot Experiments: Five physical tabletop tasks deployed on a Franka Research 3 robot arm, focusing on challenges like small-object alignment (cube stacking), tool use (pushing/pulling with sticks and hooks), and container manipulation (pouring from a cup).
Key Insights and Ideas
- Blind Optimization Can Hurt Performance: One of the paper's most striking findings is that optimizing only the planning side or performing naive joint optimization without failure attribution can actually degrade performance. Without proper credit assignment, the learning signal becomes contaminated.
- Grasping Requires Task Semantics: Geometric stability alone does not guarantee task success. A grasp must be task-compatible. GTP-FA successfully leverages VLMs to understand constraints (e.g., "don't block the hook tip") without forcing the VLM to predict precise, continuous 6-DoF coordinates.
- Smoothing Unseen Grasps: By utilizing a nearest-neighbor approach in the diagnostic embedding space ($E$), the framework successfully mitigates the brittleness of standard discriminators when encountering novel grasp configurations.
Key Results
GTP-FA was paired with five diverse base learners—PPO, SAC, Behavior Cloning (BC), Diffusion Policy (DP), and a VLA model ($\pi_{0.5}$)—and consistently delivered substantial performance gains:
- Simulation: In ManiSkill3 tasks, GTP-FA improved the terminal success rates of BC by +54.0 percentage points (pp), DP by +25.7 pp, and the VLA baseline by +17.8 pp.
- Real-Robot: When applied to the $\pi_{0.5}$ VLA model in physical testing, the average task success rate skyrocketed from a meager 11.2% (baseline) to 76.8% (GTP-FA), demonstrating massive robustness against real-world calibration errors and object slippage.
Limitations and Future Work
While highly effective, the framework currently relies on weak supervision generated from repeated trials and post-hoc execution summaries to build its failure dataset, which can be computationally expensive. The authors note that future iterations will focus on incorporating finer-grained, real-time online feedback during execution. Additionally, they plan to scale the framework to handle more complex, multi-stage tasks and a wider variety of robotic platforms and object categories.
Summary updated