Semantic Channels as the Weak Link in Sim-to-Real 6-DoF Grasping

Robot grasping networks trained only in simulation can now transfer to real arms with no real data, provided they look only at 3D geometry. OrbitGrasp is one example. It is rotation-equivariant, meaning that when the scene rotates its predicted grasps rotate with it exactly. Trained in PyBullet, it reaches a 95.2% grasp success rate on a physical UR5. Such networks are blind to meaning, though. They cannot be told "pick the mug by its handle." The obvious fix is to paint features from a 2D vision-language model such as CLIP onto the 3D points, as Splat-MOVER does for real scanned scenes. That fix carries a hidden risk. In simulation, image features come from rendered pixels. On the real robot, they come from camera pixels. A policy trained in simulation may quietly use the image channel as a shortcut for geometric cues it should be reading from depth. The image channel would then carry the full visual reality gap into a system whose geometry used to transfer cleanly.

This proposal treats that risk as the central question. We will build an equivariant 6-DoF grasp network that receives lifted 2D semantic features under a strict contract. The features enter only as rotation-invariant per-point scalars, they are randomized and dropped out in feature space during simulated training, and they can reshape which grasp poses score highly but carry no pose information themselves. We then measure, on real hardware, whether adding semantics costs geometric transfer and whether it adds task-level success that a simple "geometric grasps plus a semantic mask" pipeline cannot match.

Core Claim

Our claim has two parts. First, naively fusing image features into a sim-trained 3D grasp policy degrades zero-shot real-world grasp success relative to the same policy without them, because the network learns to read geometry partly from appearance. Second, a factorized design avoids this degradation while still letting semantics change the chosen grasp pose. In that design, multi-view-averaged foundation features enter an equivariant point network only as invariant scalar inputs, and the network is trained with feature-space randomization and semantic dropout.

The second part is what separates the design from a modular pipeline. Because the semantic scalars are inputs to the equivariant network rather than a post-hoc filter, they can change which approach direction wins. An example is approaching a mug from the side so the handle stays free. The output still transforms correctly under rotation.

Each of the four directions plays a distinct role. Rotation sensitivity supplies the mechanism: equivariant geometry is what currently transfers, so we preserve it. Cross-modal fusion is the object of study. Domain randomization is the training regime, extended from pixels into feature space. Projection-based information loss is a constraint and an evaluation lens. We keep raw 3D points rather than top-down heightmaps because heightmap systems such as CLIPort and Equivariant Transporter Network cannot express side grasps or reason about overhangs. Our test scenes deliberately include objects on shelves and under racks.

Why the gap is real

Prior work leaves this specific question open, and each paper bears on it differently.

Three works motivate the problem. OrbitGrasp shows that equivariant geometry transfers zero-shot, but it uses only point positions and surface normals. Splat-MOVER distills CLIP and affordance features into 3D Gaussians and grasps a drill by its handle 95% of the time, but it rebuilds each real scene by multi-view scanning and never trains in simulation. CLIPort fuses CLIP with a spatial stream but relies on 179 real demonstrations and planar heightmaps.

GL-MSDA is the clearest evidence that the RGB and depth reality gaps differ. It splits the two branches precisely because their noise differs, and it lifts real success only from 40% to 62.67%. It also needs unlabeled real target data and stays planar.

Countering Multi-modal Representation Collapse warns that one modality can dominate fusion. Our shortcut hypothesis is a sim-to-real version of that failure.

Several works supply method. The equivariant backbone comes from OrbitGrasp. The idea that appearance randomization pushes a network toward geometry comes from (CAD)²RL. The parameter-free local geometric encoding of GBlobs, which gave large zero-shot cross-sensor gains, informs a cheaper invariant geometric input we test as an ablation.

One paper constrains the design. Fully-Geometric Cross-Attention shows that injecting raw coordinates into fusion breaks frame consistency. This is why semantic features must never be concatenated with absolute coordinates.

Research Program

Semantic field. We render or capture several RGB views. We run a frozen 2D model (CLIP and a part-affordance model in the style used by Splat-MOVER), back-project the per-pixel features onto the fused point cloud, and average each point's features across views. This yields a per-point vector $s_p$.

Invariance property. When the whole scene and its cameras rotate rigidly together, the images do not change, so $s_p$ is exactly invariant. When an object rotates relative to the cameras, $s_p$ changes, and multi-view averaging only partly suppresses this. We will measure the residual variation rather than assume it away.

Network input. The language instruction and $s_p$ are compressed to a few scalar channels per point. One example is a similarity map between the instruction embedding and $s_p$. These channels are appended as type-0 features, the rotation-invariant scalar inputs of an equivariant network, to an OrbitGrasp-style backbone. Any rotation-invariant scalar may enter such a network without breaking equivariance. Semantics can therefore shift the predicted grasp-quality function over orientations, while every predicted pose still rotates with the scene.

Simulated training. We use the 343 object meshes from OrbitGrasp's PyBullet setup, extended with part labels on objects that have handles or functional parts. Task labels come from these part annotations. Randomization operates at three levels. At the pixel level, we randomize textures, lighting, and materials before the 2D model runs. At the depth level, we add sensor noise: dropouts, edge bleed, and missing returns. At the feature level, we add Gaussian noise to the semantic scalars, swap them between points of the same part, and drop the whole channel for a random fraction of training scenes.

Channel dropout matters for two reasons. It forces the geometric stream to stay sufficient on its own. It also gives a principled fallback when real features look out of distribution.

Diagnosis. We will measure shortcut reliance directly. On real scenes, we compare geometric grasp success when the semantic channel is replaced by a constant against success with the true features. If success with true features is lower, the semantic channel is carrying reality gap into geometry.

Baselines

The naive reference is OrbitGrasp itself, retrained in our pipeline with no semantic input. It answers whether semantics cost anything geometrically. The file reports 96.7% grasp success in single-view Pile scenes in simulation, 99.2% declutter rate in Packed scenes, and 95.2% real grasp success on a UR5, all from OrbitGrasp. Our method must not fall more than a few points below our own reproduction of these numbers on purely geometric tasks.

The strongest competitor is a modular pipeline. OrbitGrasp proposes its top-K grasps from geometry alone. The same lifted semantic field then filters and reranks them by how much the instruction matches the region near each grasp contact. It uses identical simulated training data, the identical 2D models, and the identical backbone and training compute. Its only tuning freedom, K and the scoring threshold, gets the same search budget as our method's dropout rate and noise scale. Beating it means higher real task-success rate on task-oriented instructions, defined as a successful lift with the gripper on the instructed part. We will report this separately for scenes where the correct grasp requires a non-default approach direction.

We are not aware of a reported number for task-oriented 6-DoF grasping in the file's benchmarks. No reported result was found on Lacuna for this benchmark, so we establish all task numbers ourselves.

A naive-fusion baseline concatenates raw RGB and full CLIP feature vectors to every point and trains with only pixel-level randomization. This is the design our shortcut hypothesis predicts will degrade.

A planar baseline is a CLIPort-style heightmap policy trained in the same simulator with the same visual randomization. It quantifies projection loss on our shelf and overhang scenes.

Two ablations isolate the key ingredients. The first removes feature-space randomization and channel dropout while keeping the invariant-scalar interface. The second keeps everything but uses a non-equivariant PointNet++ with rotation augmentation. Its geometric weakness is already documented in OrbitGrasp's ablation, where it reached 60.5% grasp success in Pile against 96.7% for the equivariant model. Here we test whether that gap widens once semantics are added.

Why not just…?

Why not just filter OrbitGrasp's grasps with a 3D semantic mask? This is the real threat, and the experiments are built to settle it. Filtering cannot introduce a grasp that geometry ranked below the cutoff. It wins when the instructed part also hosts a geometrically good grasp. It should lose only when the task requires a less stable approach the geometric ranking pushes down, such as a side grasp on a mug handle when a top grasp on the rim scores higher. We stratify real trials by this property. If the modular pipeline matches us on the hard stratum too, we will conclude that end-to-end fusion is unnecessary and report the shortcut diagnosis as the main finding.

Why not just use rotation augmentation on a standard fused network? OrbitGrasp's own ablation shows an augmented non-equivariant network losing more than 35 points of simulated grasp success. Effective Rotation-Invariant Point CNN with Spherical Harmonics Kernels notes that augmentation gives no guarantee on unseen poses. Our PointNet++ ablation checks whether that still holds once a semantic channel is present.

Why not just use heightmaps with domain randomization, as CLIPort-style systems would? Heightmaps collapse the vertical axis. UVTR shows that keeping height in its voxel grid adds 4.2 NDS, a nuScenes detection score, over a collapsed grid. Heightmaps also restrict actions to planar grasps. The planar baseline on shelf and overhang scenes will show whether this matters in practice for our tasks rather than in principle.

Why not just rely on pixel-level randomization alone? It may already make frozen foundation features stable enough across simulation and reality. The first ablation tests exactly this. If removing feature-space randomization does not hurt real transfer, we will drop it and say so.

Proposed Experiments

Simulation. We use OrbitGrasp's Pile and Packed protocols for geometric grasping. We add a task-oriented split with part-labeled objects and language instructions, including shelf and overhang configurations.

Real hardware. We run zero-shot trials on a UR5 with the same object categories, unseen instances, and both default and non-default-approach tasks. All methods get the same number of trials per scene.

Metrics. We report grasp success rate, declutter rate, task-success rate, and inference latency. Latency matters because OrbitGrasp's backbone takes about 1.5 seconds per inference.

Decisive checks. There are three:

  1. Shortcut test. Naive fusion should show lower real geometric grasp success than the no-semantics model, while ours should show none.
  2. Hard-stratum test. Our method should beat the modular filter on non-default-approach tasks.
  3. Invariance measurement. We measure semantic-field variation when objects rotate relative to the cameras, and correlate it with per-trial failures.

If the first check fails, meaning naive fusion transfers fine, the central hypothesis is wrong. In that case the project reduces to an engineering comparison against the modular pipeline.