TL;DR
Current grasp-feasibility predictors often achieve near-perfect scores by exploiting "shortcuts" in grasp poses rather than actually reasoning about scene geometry. This paper introduces a planner-verified counterfactual audit that exposes these failures across eight state-of-the-art models and demonstrates that explicit clearance-based representations are necessary to truly capture collision logic.
What Problem Does This Paper Address?
In robotic manipulation, determining if a specific grasp is "feasible"—meaning a collision-free path exists to reach it—is a computationally expensive task for traditional motion planners. To speed things up, researchers have developed "feasibility predictors": neural networks that look at a scene (as a point cloud, voxels, or depth map) and quickly score a grasp's likelihood of success.
While these models report stellar accuracy on standard benchmarks, there's a catch. Standard datasets rarely include "hard" examples where a tiny change in geometry (like a small box blocking the path) flips a grasp from feasible to infeasible. Consequently, models often learn to ignore the scene geometry entirely, relying instead on the grasp pose itself as a shortcut. If a pose looks "good" based on training data, the model calls it feasible, even if a new obstacle is clearly in the way.
Methods
The authors propose a Planner-Verified Counterfactual (CF) test. The process is simple but rigorous:
- Take a grasp that a motion planner has already verified as feasible.
- Hold the grasp pose fixed and insert a solid obstacle (like a small cube) directly into the robot's approach corridor.
- Re-run the motion planner to confirm the label has now flipped to infeasible.
- Feed this new scene to the predictor and see if its score changes.
Beyond auditing, the authors propose a Clearance-Aware Tokenization method. Instead of letting a black-box encoder process the whole scene, they slice the approach corridor into axial slabs and explicitly extract the minimum lateral clearance for each. This forces the model to "read" the specific geometry that a motion planner checks.
Datasets
The study utilizes several key datasets and introduces a new auditing benchmark:
- GraspNet-1B: The primary source for real-world cluttered tabletop scenes.
- GraspNet-1B Counterfactual Set: A new, publicly released benchmark containing 1,065 planner-verified counterfactual pairs, including various obstacle families (spheres, L-shapes, pillars).
- ACRONYM & PR2 Kitchen: Used to audit models (SceneCollisionNet and PIGINet) on their original training environments.
- panda_3d_4: Used for auditing the Geometric Reasoning Network (GRN).
Key Insights and Ideas
- The "Pose Shortcut": In natural datasets, the grasp pose is often enough to predict feasibility. Models "saturate" benchmarks without ever learning to detect collisions.
- Resolution Blindness: Models fail at the resolution of their internal representation. For example, a model using 12.5 cm voxels is physically incapable of "seeing" a 6 cm gap, leading to persistent false positives.
- Input vs. Extraction: The collision signal is present in the raw input (point clouds), but standard binary supervision (feasible/infeasible) doesn't force the model to extract it.
- Structural Cures: Explicitly supervising for "clearance" (the distance to the nearest obstacle) rather than just feasibility allows models to recover the collision term even from unmodified benchmarks.
Key Results
- Total Collapse on CF Test: Eight state-of-the-art predictors that scored 0.92–0.99 AUROC on standard benchmarks dropped to chance levels (0.47–0.50) when faced with counterfactual obstacles.
- Widespread Failure: This failure was consistent across four different motion planners, two different robot arms (Franka Panda and xArm6), and various obstacle types, including real-world objects.
- Diagnostic Success: The proposed clearance-aware tokenization reached a 0.983 CF-pair AUROC when trained on the unmodified benchmark, proving that the necessary information is there if the architecture is designed to look for it.
- Resolution Limits: Models like GRN, which use bounding boxes, could detect the presence of an object but were blind to its actual shape (e.g., failing to distinguish a box from a leaning pole that actually causes a collision).
Limitations and Future Work
The authors acknowledge that while their clearance-aware representation is a powerful diagnostic tool, it is not yet a "silver bullet" for deployment. It remains placement-bound, meaning its accuracy degrades if obstacles are placed in locations far outside its training distribution.
The primary open challenge is developing a placement-invariant clearance representation—a model that can reason about collisions anywhere in the workspace, not just within a pre-defined "approach corridor." Additionally, the current study focuses on single-arm setups; extending this rigorous auditing to multi-arm coordination or more complex kinematic chains is a promising next step.
Summary updated