Auditing Grasp Feasibility Predictors with Planner-Verified Counterfactuals

Sajid Ansari, Antony Thomas, Girish VarmaarXiv 2026 · arXiv

TL;DR

Current grasp-feasibility predictors often achieve near-perfect scores by exploiting "shortcuts" in grasp poses rather than actually reasoning about scene geometry. This paper introduces a planner-verified counterfactual audit that exposes these failures across eight state-of-the-art models and demonstrates that explicit clearance-based representations are necessary to truly capture collision logic.

What Problem Does This Paper Address?

In robotic manipulation, determining if a specific grasp is "feasible"—meaning a collision-free path exists to reach it—is a computationally expensive task for traditional motion planners. To speed things up, researchers have developed "feasibility predictors": neural networks that look at a scene (as a point cloud, voxels, or depth map) and quickly score a grasp's likelihood of success.

While these models report stellar accuracy on standard benchmarks, there's a catch. Standard datasets rarely include "hard" examples where a tiny change in geometry (like a small box blocking the path) flips a grasp from feasible to infeasible. Consequently, models often learn to ignore the scene geometry entirely, relying instead on the grasp pose itself as a shortcut. If a pose looks "good" based on training data, the model calls it feasible, even if a new obstacle is clearly in the way.

Methods

The authors propose a Planner-Verified Counterfactual (CF) test. The process is simple but rigorous:

  1. Take a grasp that a motion planner has already verified as feasible.
  2. Hold the grasp pose fixed and insert a solid obstacle (like a small cube) directly into the robot's approach corridor.
  3. Re-run the motion planner to confirm the label has now flipped to infeasible.
  4. Feed this new scene to the predictor and see if its score changes.
Figure 1. The counterfactual test. (left) A feasible grasp. (right) The same grasp becomes infeasible once a cube (red) is inserted in its approach path; points sampled from the cube’s surface are added to the point-cloud input. The classifier (GraspFC-PTX) still predicts p(feasible) = 0.86 in both scenes. Over 1,065 such pairs, its CF-pair AUROC is 0.498 (chance = 0.5).

Beyond auditing, the authors propose a Clearance-Aware Tokenization method. Instead of letting a black-box encoder process the whole scene, they slice the approach corridor into axial slabs and explicitly extract the minimum lateral clearance for each. This forces the model to "read" the specific geometry that a motion planner checks.

Datasets

The study utilizes several key datasets and introduces a new auditing benchmark:

  • GraspNet-1B: The primary source for real-world cluttered tabletop scenes.
  • GraspNet-1B Counterfactual Set: A new, publicly released benchmark containing 1,065 planner-verified counterfactual pairs, including various obstacle families (spheres, L-shapes, pillars).
  • ACRONYM & PR2 Kitchen: Used to audit models (SceneCollisionNet and PIGINet) on their original training environments.
  • panda_3d_4: Used for auditing the Geometric Reasoning Network (GRN).

Key Insights and Ideas

  • The "Pose Shortcut": In natural datasets, the grasp pose is often enough to predict feasibility. Models "saturate" benchmarks without ever learning to detect collisions.
  • Resolution Blindness: Models fail at the resolution of their internal representation. For example, a model using 12.5 cm voxels is physically incapable of "seeing" a 6 cm gap, leading to persistent false positives.
  • Input vs. Extraction: The collision signal is present in the raw input (point clouds), but standard binary supervision (feasible/infeasible) doesn't force the model to extract it.
  • Structural Cures: Explicitly supervising for "clearance" (the distance to the nearest obstacle) rather than just feasibility allows models to recover the collision term even from unmodified benchmarks.
Figure 2. The counterfactual protocol on a GraspNet-1B scene (deployed GraspFC-PTX). One grasp is held fixed; each of three obstacles (a floating corridor box, a resting pillar, a transit-path box) makes the planner relabel it infeasible (red), while the clear grasp stays feasible (green). The model returns the same p(feasible), 0.975 to 0.976, for all four, so its score does not sep- arate the clear scene from any of the three blocked ones.

Key Results

  • Total Collapse on CF Test: Eight state-of-the-art predictors that scored 0.92–0.99 AUROC on standard benchmarks dropped to chance levels (0.47–0.50) when faced with counterfactual obstacles.
  • Widespread Failure: This failure was consistent across four different motion planners, two different robot arms (Franka Panda and xArm6), and various obstacle types, including real-world objects.
  • Diagnostic Success: The proposed clearance-aware tokenization reached a 0.983 CF-pair AUROC when trained on the unmodified benchmark, proving that the necessary information is there if the architecture is designed to look for it.
  • Resolution Limits: Models like GRN, which use bounding boxes, could detect the presence of an object but were blind to its actual shape (e.g., failing to distinguish a box from a leaning pole that actually causes a collision).
Figure 6. GRN detects the bounding box of an object but fails to resolve sub-bounding-box shapes like a leaning pole.

Limitations and Future Work

The authors acknowledge that while their clearance-aware representation is a powerful diagnostic tool, it is not yet a "silver bullet" for deployment. It remains placement-bound, meaning its accuracy degrades if obstacles are placed in locations far outside its training distribution.

The primary open challenge is developing a placement-invariant clearance representation—a model that can reason about collisions anywhere in the workspace, not just within a pre-defined "approach corridor." Additionally, the current study focuses on single-arm setups; extending this rigorous auditing to multi-arm coordination or more complex kinematic chains is a promising next step.

Summary updated

Abstract

Grasp-feasibility predictors report near-perfect accuracy on Task and Motion Planning benchmarks, but high accuracy on these benchmarks does not verify that a predictor uses the collision geometry that the motion planner checks. We propose a planner-verified counterfactual (CF) test: holding the grasp pose fixed, we insert a solid obstacle into the approach corridor, add its points to the model input, and let a planner re-derive the label, which flips from feasible to infeasible. Across eight predictors spanning point clouds, voxels, depth, and object tokens, CF-pair AUROC falls from high natural AUROC (0.920-0.996) to chance (0.470-0.504), and the flip holds across four planners, two arms, and a real-object obstacle. Predictors fail at different resolutions, from a bounding box to a 12.5 cm voxel. In the point-cloud models we dissect, the obstacle either moves the representation in a direction unrelated to whether it blocks the grasp or is not registered at all. A clearance-aware read of the corridor points, trained on the unmodified benchmark with no counterfactual scenes, reaches 0.983 CF-pair AUROC, but it remains bound to its trained obstacle placements; a placement-invariant clearance representation is still open. We release the counterfactual labels, metrics, datasheet, and relabeling pipeline at https://github.com/codewithsajid/graspfc-cf.
BibTeX
@misc{ansari2026auditing,
  author = {Sajid Ansari and Antony Thomas and Girish Varma},
  title = {{Auditing Grasp Feasibility Predictors with Planner-Verified Counterfactuals}},
  year = {2026},
  eprint = {2611.09130},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09130},
}
Figure