Grasp-Then-Plan with Failure Attribution: A Closed Two-Stage Framework for Precise and Generalizable Robotic Manipulation

Jiahao Xu, Peiyuan Wang, Hanzhuo Zhang, Zihao Yu, Tianyu Fu, Hao Chen, Xuanhao Xiang, Jianbo Yu, Chenchen Fu, Wanyuan WangarXiv 2026 · arXiv

TL;DR

GTP-FA (Grasp-Then-Plan with Failure Attribution) is a closed-loop framework that decouples robotic manipulation into distinct grasping and planning stages while intelligently diagnosing execution failures. By accurately attributing whether a failure stems from a faulty grasp or a downstream planning error, the system routes optimization pressure to the correct module, significantly boosting the success rates of reinforcement learning, imitation learning, and vision-language-action (VLA) models.

What Problem Does This Paper Address?

In long-horizon robotic manipulation, a task's success is heavily dictated by the initial grasp. For instance, if a robot picks up a tool by its functional end rather than its handle, even the most advanced motion planning policy will inevitably fail later in the execution.

This creates a challenging "credit assignment" problem due to the tight coupling between grasping and planning. When a failure occurs late in a trajectory, standard end-to-end learning algorithms often misattribute the error to the downstream planning policy rather than the fundamentally flawed initial grasp. This misattribution contaminates training data and severely weakens the robot's ability to generalize to new scenarios. While separating the pipeline into "grasp-then-plan" stages helps, traditional methods still fail to communicate downstream execution realities back to the grasp generator.

Methods

To close this loop, the authors introduce GTP-FA, which operates as an iterative execute-diagnose-update pipeline.

Figure 1: Overview of GTP-FA. Given a task description, GTP-FA performs task-aware grasp optimization, executes the selected grasp, diagnoses the resulting outcome, and routes optimization signals to either the grasp side or the planning side in a closed-loop execute–diagnose–update pipeline. The optimized grasp-then-plan interface is then transferred to real-robot deployment.

The framework relies on three core components:

  1. Failure Attribution Discriminator ($D$): A lightweight model that categorizes execution failures into three distinct modes: task-irrelevant grasp contact (FM-G1), unstable grasping/slippage (FM-G2), or downstream planning insufficiency (FM-P). Training labels are generated automatically via weak supervision from repeated trials under fixed grasp conditions.
  2. Grasp-Conditioned Diagnostic Representation Space ($E$): To ensure the discriminator generalizes to entirely unseen test-time grasps, this component maps post-grasp initialization states into an embedding space using a supervised contrastive objective. It enforces a local-consistency prior: grasps that result in similar physical states should share similar failure distributions.
  3. Diagnosis-Driven Bidirectional Optimization: The system calculates responsibility weights based on the fused outputs of $D$ and $E$.
  4. If the failure is grasp-dominant, the grasp selection module is updated by injecting task-level semantic priors (via a Vision-Language Model) and diagnostic risk penalties to suppress unstable candidate grasps.
  5. If the failure is planning-dominant, the downstream policy is refined using "hard-planning" sample mining, training distribution reshaping, and responsibility-weighted loss functions.
Figure 2: Failure attribution and diagnostic signal generation. Given a task instruction, visual obser- vation, candidate grasp, and execution summary, the discriminator D performs post-hoc diagnosis for a single execution and outputs a probability distribution over three failure modes. To improve attribution stability for unseen grasps, the system uses the grasp-conditioned representation space E to construct a neighborhood diagnostic prior, which is fused with the output of D to produce a reliable diagnostic signal for grasp-side optimization and planning-side downstream policy learning.

Datasets

The authors evaluated GTP-FA across both simulated and physical environments:

  • Simulation: Eight long-horizon manipulation tasks from the ManiSkill3 benchmark, including StackCube, PokeCube, LiftPegUpright, PlaceSphere, PickCube, PushCube, PullCube, and PullCubeTool.
  • Real-Robot Experiments: Five physical tabletop tasks deployed on a Franka Research 3 robot arm, focusing on challenges like small-object alignment (cube stacking), tool use (pushing/pulling with sticks and hooks), and container manipulation (pouring from a cup).

Key Insights and Ideas

  • Blind Optimization Can Hurt Performance: One of the paper's most striking findings is that optimizing only the planning side or performing naive joint optimization without failure attribution can actually degrade performance. Without proper credit assignment, the learning signal becomes contaminated.
  • Grasping Requires Task Semantics: Geometric stability alone does not guarantee task success. A grasp must be task-compatible. GTP-FA successfully leverages VLMs to understand constraints (e.g., "don't block the hook tip") without forcing the VLM to predict precise, continuous 6-DoF coordinates.
  • Smoothing Unseen Grasps: By utilizing a nearest-neighbor approach in the diagnostic embedding space ($E$), the framework successfully mitigates the brittleness of standard discriminators when encountering novel grasp configurations.

Key Results

GTP-FA was paired with five diverse base learners—PPO, SAC, Behavior Cloning (BC), Diffusion Policy (DP), and a VLA model ($\pi_{0.5}$)—and consistently delivered substantial performance gains:

  • Simulation: In ManiSkill3 tasks, GTP-FA improved the terminal success rates of BC by +54.0 percentage points (pp), DP by +25.7 pp, and the VLA baseline by +17.8 pp.
  • Real-Robot: When applied to the $\pi_{0.5}$ VLA model in physical testing, the average task success rate skyrocketed from a meager 11.2% (baseline) to 76.8% (GTP-FA), demonstrating massive robustness against real-world calibration errors and object slippage.

Limitations and Future Work

While highly effective, the framework currently relies on weak supervision generated from repeated trials and post-hoc execution summaries to build its failure dataset, which can be computationally expensive. The authors note that future iterations will focus on incorporating finer-grained, real-time online feedback during execution. Additionally, they plan to scale the framework to handle more complex, multi-stage tasks and a wider variety of robotic platforms and object categories.

Summary updated

Abstract

In robotic manipulation, the tight coupling between grasping and motion planning often obscures the true source of failure, leading to inefficient trial-and-error. To enable efficient long-horizon manipulation, we propose GTP-FA (Grasp-Then-Plan with Failure Attribution), a task-oriented two-stage grasp-then-plan framework that generates grasp candidates and performs downstream motion planning conditioned on the selected grasp. Given a failed manipulation trajectory, we learn a failure attribution model that generalizes to unseen grasps and produces a stable distribution over failure modes for diagnosis-guided optimization. Based on these attribution results, we then optimize both modules in a diagnosis-driven manner: on the grasping side, we inject task-level priors and risk penalties into grasp candidate scoring and optimization to suppress unstable or task-incompatible grasps; on the planning side, we target high-risk initial states through data collection and fine-tuning to address genuine planning bottlenecks. We evaluate the proposed framework in both simulation and real-robot experiments, and show that GTP-FA improves the corresponding base learners across RL, IL, diffusion-policy, and VLA-based settings, achieving substantially higher overall task success rates.
BibTeX
@misc{xu2026grasp,
  author = {Jiahao Xu and Peiyuan Wang and Hanzhuo Zhang and Zihao Yu and Tianyu Fu and Hao Chen and Xuanhao Xiang and Jianbo Yu and Chenchen Fu and Wanyuan Wang},
  title = {{Grasp-Then-Plan with Failure Attribution: A Closed Two-Stage Framework for Precise and Generalizable Robotic Manipulation}},
  year = {2026},
  eprint = {2606.03385},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.03385},
}
Figure