Letting a Local Grasp Model See Obstacles Without Seeing Objects

The robot grasp detectors that transfer best to never-before-seen objects deliberately look at very little of the scene. They cut out a small ball of 3D points around a candidate grasp and decide how to grasp from that local shape alone. This transfers because edges, rims and handles look alike across object categories. The price is that such a detector cannot see an obstacle sitting just outside its crop. Detectors that look at the whole scene handle clutter better but tend to memorize the training objects. We propose a middle path. The local grasp model keeps its small crop, and it also receives a second input that describes the rest of the scene only as occupied or empty space, expressed in the gripper's own coordinate frame. That second input carries no color, no learned object features and no sense of which object is where. It only says whether geometry lies along the path the fingers will sweep.

The core bet is that global context hurts generalization because it leaks object identity, not because it is global. If that is right, a context stream stripped down to identity-free occupancy should recover the collision awareness of scene-level models while keeping the cross-category transfer of local ones. The gripper frame also solves an efficiency problem. Factorized attention schemes that make whole-image context cheap need orthogonal axes, which raw point clouds lack. A gripper has three natural axes: the direction it approaches from, the direction its fingers close, and the finger height. Attention can run along those axes.

Where this sits in prior work

Two camps in 6-DoF grasp detection (predicting a full 3D position and orientation for a two-finger gripper) motivate the proposal. The local camp is represented by FlexLoG, Rethinking 6-DoF grasp detection: A flexible framework for high-quality grasping. FlexLoG splits detection into a guidance module that picks where to look and a local grasp model that encodes a spherical crop with a light point-based network. It reports an 18–23% average-precision gain over HGGD on novel objects. Also in the local camp is RNGNet, Region-aware Grasp Framework with Normalized Grasp Space for Efficient 6-DoF Grasping, which normalizes depth-adaptive patches into a canonical grasp space and runs at 50 frames per second. Both papers say plainly that the crop makes the model blind to collisions outside the patch. FlexLoG pushes that burden onto a separate collision head.

The context camp widens what the model sees. 6-DoF Grasp Detection in Clutter with Enhanced Receptive Field and Graspable Balance Sampling uses four hand-set cylindrical grouping radii aligned with the gripper. Its authors found the enlarged field brought in enough irrelevant points that they needed an extra attention module to suppress them. GraNet spreads information across nearest-neighbor graphs over several hops at scene, object and grasp-point levels. Both widen the region of features the model sees, including features that encode what an object is. Our proposal differs on exactly this point. We widen the region of geometry the model can query, and we withhold appearance and semantic features from that wider region.

On the method side, we reuse the factorization idea from CCNet: Criss-Cross Attention for Semantic Segmentation. CCNet attends along image rows and columns and repeats the pass so that every position reaches every other. It uses 11x less GPU memory than full Non-Local Neural Networks, which compare every position with every other. The research file notes as an open problem that criss-cross attention needs a grid with axes, which scene point clouds do not have. The candidate grasp pose provides one.

Core Claim

Given a candidate grasp pose $g$, we build two inputs. The first is the usual local crop, encoded as in FlexLoG. The second is a coarse voxel grid of $a \times b \times c$ cells laid out along the gripper's approach, closing and height axes. The grid extends well beyond the crop, covering the full approach corridor and the space behind the fingers.

Each cell holds three things. The first is an occupancy count from the observed points. The second is a free-space flag obtained by casting rays from the camera, so the grid separates "seen to be empty" from "never observed." The third is the cell's gripper-frame coordinates. Nothing else goes in.

Recurrent criss-cross attention runs along the three gripper axes, with two passes as in CCNet. This costs $O(G(a+b+c))$ for $G = abc$ cells, instead of the $O(G^2)$ of full attention. After these passes, every cell has seen every other cell. The local features then read from this grid through a narrow gate:

$$ h = f_{\text{loc}}(P_g) + \sigma(w^\top z_g)\, U z_g , $$

Here $f_{\text{loc}}(P_g)$ is the local-crop embedding of the points $P_g$ inside the crop. The term $z_g$ is a low-dimensional summary (we propose 16 dimensions) pooled from the context grid. The vector $w$ and matrix $U$ are learned, and $\sigma$ is the logistic function. The narrow summary and the gate limit how much context can reshape the local representation. The pooled summary also feeds the collision and quality predictions directly.

Taken together, the design makes a falsifiable claim. Detection on the GraspNet-1Billion novel split should stay at local-model levels, and collision failures in dense clutter should fall to or below scene-level levels. Adding semantic features to the context grid should measurably hurt the novel split.

Research Program

The first stage is to build the system on GraspNet-1Billion (GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping). This benchmark has 190 cluttered scenes, 88 objects and over a billion analytically labeled grasps, split into seen, similar and novel objects. The local encoder is pretrained on the 6.5-million-region Regional Grasp-Centric Dataset released with FlexLoG. The context stream is then trained jointly with it on full scenes, using the guidance module from FlexLoG to propose candidates.

The second stage is analysis. It probes whether the context summary $z_g$ encodes object identity. We train a linear classifier to predict the object instance or category from $z_g$, compare against the same probe on context summaries built from full backbone features, and relate probe accuracy to the drop from seen to novel AP. This is the mechanistic test of the identity-leakage hypothesis. We also measure how often the gate opens, as a function of clutter density and of distance to the nearest out-of-crop obstacle.

The third stage is a small physical study. It follows the bin-clearing protocols used in FlexLoG and GraNet, and we report the rate of executed grasps that fail through collision separately from grasps that fail through slip.

Baselines

The naive reference point is the local grasp model alone, with no collision reasoning beyond its own crop. This is FlexLoG's local model with its collision head disabled, trained on identical data. It fixes the floor on clutter performance and the ceiling that locality allows on novel transfer.

The strongest existing competitor on this exact question is full FlexLoG, which pairs the local model with a guidance module and a dedicated collision head. It already achieves strong novel-object transfer with some collision awareness, and it reports 95% real-world success in clutter. The file gives FlexLoG's novel-split result only as an 18–23% relative AP improvement over HGGD. No absolute FlexLoG novel AP is reported on Lacuna, so we will rerun the released configuration on the same splits, with the same candidate budget and the same number of training iterations as our model. Beating it means higher AP on the novel split and on a dense-clutter subset of scenes, together with a lower rate of top-ranked grasps rejected for scene collision. A lead of one or two AP points counts only if it holds across three seeds.

The strongest context-camp baseline is the multi-radii cylinder grouping network with graspable balance sampling. It reports novel $AP_{0.4}$ rising from 14.23% to 20.23% and a 10% AP gain over GSNet (Graspness Discovery in Clutter for Fast and Accurate Grasp Detection (GSNet)). Its balance sampling depends on a pretrained instance segmenter, so we run it both with that segmenter and with farthest-point sampling. The second run separates the sampling gain from the receptive-field gain. GraNet and RNGNet are included as further reference points under their published settings where code permits.

Three ablations isolate the key ingredient, and all keep parameter count and FLOPs matched. The first feeds the context grid learned backbone features, including color, instead of occupancy only; this tests identity leakage directly. The second replaces gripper-frame criss-cross attention with the same grid built in the camera frame; this tests whether the gripper's axes matter. The third removes the free-space ray channel; this tests whether knowing "unobserved" versus "empty" matters.

Why not just…?

Why not just run an analytic collision check of each predicted grasp against the scene point cloud? This is the most serious threat. GraspNet-style pipelines already filter grasps with a model-free collision detector, and it costs nothing to train. Its weakness is that on a single-view cloud it treats unobserved space as empty, so it approves grasps whose fingers sweep into occluded geometry. It also cannot judge quality effects that are not strict collisions, such as a neighboring object pinning the target. We will run the local model plus the analytic filter as a baseline and break failures down by whether the colliding geometry was observed or occluded. If the filter matches our model on both kinds of failure, the learned context stream is not justified, and we will report that outcome as the finding.

Why not just enlarge the crop, or use several fixed radii? A larger crop brings object shape back into the local features, which is exactly what the local camp avoids. The multi-radii paper itself found that the enlarged field added noise that required an extra suppression module. We will sweep crop radius on the local model alone. If AP on the novel split falls as the radius grows while collisions drop, that tradeoff curve is the one our design aims to escape. If a single larger radius dominates at every density, our idea fails.

Why not just add standard global attention or a pooled scene vector? Full non-local attention over tens of thousands of scene points is memory-bound, which is the problem CCNet was built to solve. A pooled scene vector is cheap but carries whatever the backbone learned, including object identity. The first ablation above is exactly this alternative at matched cost, so the experiment will show whether the occupancy-only restriction, rather than globality, is what matters. Compact Generalized Non-local Network also reports that plain non-local blocks focus on the main subject and miss interactions with surrounding objects. Those surrounding interactions are the ones a grasp needs.

Decisive Checks

Three results would settle the proposal. First, on GraspNet-1Billion the full model should match the local-only model's novel-split AP within seed variance, while cutting the share of collision-rejected top-50 grasps in the densest third of scenes by a clear margin relative to FlexLoG. Second, the semantic-context ablation should raise probe accuracy for object identity and lower novel-split AP, while leaving seen-split AP flat or higher. That pattern is the signature of identity leakage, and its absence would falsify our explanation even if the model still performed well. Third, against the analytic collision filter, our advantage should be concentrated in failures caused by occluded geometry. If none of the gain comes from occluded cases, the analytic filter is the better engineering choice.