Cross-View Depth Disagreement as a Signal for Grasping Glass

Depth cameras lie about glass. An infrared depth sensor pointed at a wine glass usually returns either nothing or the distance to the table behind it, while reporting the surrounding mugs, shelf, and tabletop accurately. Current systems handle this in one of two ways. They detect the glass in a color image, erase the depth there, and fill it in, a chain of separate steps in which a missed mask silently corrupts every later step. Or they throw the depth camera away and reconstruct everything from color images. This proposal starts from a simple physical observation. When the same scene is seen from several positions, depth readings on opaque surfaces agree with one another once mapped into a common 3D frame. Depth readings on glass do not, because each viewpoint sees a different patch of background through, or around, the refracting surface. That disagreement can be measured without any labels. We propose to compute it per 3D cell and use it to decide, cell by cell, how much a learned grasping network should rely on depth versus color features.

The concrete system fuses RGB-D views, which may come from a wrist camera, a second robot arm, or a fixed overhead camera, into a 3D grid of features. A small gating network reads the cross-view depth disagreement in each cell and down-weights depth where it is untrustworthy. A grasp network is trained end to end on the gated volume. The idea ties together the four directions in a specific way. Extra viewpoints, the standard cure for objects hidden behind other objects, are reused as a lie detector for the depth modality. The gate is a soft, inspectable map rather than a hard mask, so an upstream mistake degrades a confidence instead of deleting geometry. The work also targets a known failure of 2D-to-3D feature fusion: it assumes the 3D point and the 2D pixel come from the same physical surface, and that assumption breaks on refractive objects.

Core Claim

Cross-view inconsistency of active depth is a reliable, label-free indicator of non-Lambertian corruption. Here "non-Lambertian" means surfaces such as glass and polished metal that do not scatter light evenly. Using this indicator to gate the depth stream lets a multi-view RGB-D grasp network keep metric depth where it is valid and fall back on color-based geometry where it is not. The claim is falsifiable at two levels. First, the disagreement statistic should separate transparent from opaque regions on real data before any learning. Second, the gated network should beat both the best RGB-only multi-view grasping method and RGB-D pipelines that remove depth using a single-view transparency mask, with the largest margins in cluttered and occluded scenes.

Position Relative to Prior Work

Two lines of transparent-object work set up the tension this proposal resolves. The first keeps the depth camera and repairs it. ClearGrasp segments transparent pixels, predicts surface normals and boundaries from RGB, and solves a global optimization to inpaint depth. TransNet assumes given instance masks, completes depth, and regresses category-level pose. These papers motivate the proposal by showing that RGB cues recover transparent shape, and they also expose the cascade problem: both depend on an upstream mask that the rest of the pipeline cannot question. ClearGrasp further reports that regions fully enclosed by occlusion boundaries get indeterminate depth from a single view.

The second line abandons active depth. KeyPose triangulates keypoints from stereo, SimNet trains a stereo cost-volume network on synthetic data, MVTrans builds a multi-view plane-sweep volume for depth, segmentation, and pose, and GraspNeRF predicts a signed-distance grid from six RGB views and feeds it to a 3D CNN for 6-DoF grasps trained end to end. These papers supply our method. We reuse GraspNeRF's backproject-into-a-grid design and its grasp head. They also constrain it, because they show RGB-only multi-view geometry is already strong. Their shared choice is to discard depth everywhere, including on the opaque table, walls, and neighboring objects where depth is excellent and supplies metric anchoring. We keep depth and learn where to ignore it.

SceneGrasp motivates the end-to-end design: it shows that a shared latent space for shape, pose, and grasp removes cascades between modules, but only on opaque objects whose depth is clean. The cross-modal fusion literature, including PointFusion, MVX-Net, and 3DMV, attaches image features to 3D points or voxels and assumes both come from the same surface. We change the fusion rule, not the backprojection itself.

From collaborative driving perception, CoBEVT learns attention over agents and grid cells to favor unobstructed viewpoints, and Where2comm uses spatial confidence maps to decide what to share. Both learn confidence implicitly from outdoor scenes of opaque objects. Our gate instead consumes an explicit physical statistic. Whether that statistic is necessary, or whether attention would learn it anyway, is the central ablation.

Research Program

Each view $i$ supplies a color image, a depth map, and an estimated camera pose. For a grid cell at position $x$, each view in which $x$ falls inside the camera frustum produces a truncated signed distance $d_i(x)$: the distance from $x$ to the surface that view's depth reports along the camera ray, clipped to a band around the surface. Cells with no depth return contribute a "missing" flag. The disagreement statistic is the visibility-weighted spread of these votes:

$$c(x) = \frac{\sum_i w_i(x)\,\big(d_i(x)-\bar d(x)\big)^2}{\sum_i w_i(x)}, \qquad \bar d(x) = \frac{\sum_i w_i(x)\, d_i(x)}{\sum_i w_i(x)},$$

where $w_i(x)$ is the standard signed-distance fusion weight, which decays with distance from the measured surface and with grazing viewing angle. The fraction of views returning no depth at $x$ is kept as a separate channel.

A small network takes $c(x)$, the missing-depth fraction, the number of viewing cameras, and the backprojected RGB feature. It outputs a gate $g(x)\in[0,1]$, and the fused cell feature is $g(x)\,f_{\text{depth}}(x) + (1-g(x))\,f_{\text{rgb}}(x)$. Here $f_{\text{depth}}$ encodes the fused depth signed distance and $f_{\text{rgb}}$ is a GraspNeRF-style feature aggregated from color images across views. A 3D CNN then predicts a signed-distance field and grasp quality, orientation, and width, as in GraspNeRF. Training uses grasp and geometry losses only. The gate never sees transparency labels, so any alignment between the gate and glass is evidence that the statistic carries the signal.

Two confounds have to be designed out. The first is pose error. When views come from different robots, a relative pose error also inflates $c(x)$, but it does so smoothly over the whole scene rather than in object-shaped patches. We therefore first refine relative poses by aligning only cells with low disagreement, a robust iterative step, and report the gate's behavior under injected pose noise. The second is occlusion contamination. Backprojecting a foreground pixel's color feature into background cells along the same ray is a known flaw of naive lifting. We use depth-derived visibility to suppress this contamination, but only where $g(x)$ is high, because visibility computed from lying depth on glass is itself unreliable.

Training data needs depth corrupted the way real sensors corrupt it. GraspNeRF's synthetic scenes report RGB images and grasp labels, but the file does not say they include sensor-realistic depth. We will add a simulated structured-light depth channel. It ray-traces the first diffuse surface hit, so glass registers the background, and it drops returns at specular grazing angles. Because this simulator is a modeling choice, every claim is validated on real data from ClearPose and from physical grasps.

Baselines

The naive reference point fuses raw depth from all views into a signed-distance grid and runs the grasp network on it with no RGB branch and no gate. This is the VGN-style setup that GraspNeRF compares against, for which GraspNeRF reports 66.6% real single-object grasp success versus its own 88.9%. It shows how much damage lying depth does when nothing corrects it.

The strongest existing competitor is GraspNeRF itself. It is the best reported end-to-end multi-view grasping method for transparent and specular clutter, with 88.9% real single-object retrieval and 82.2% in packed clutter. It also represents the "just drop depth" answer to our question. We will retrain it on exactly our synthetic scenes, view counts, and camera placements, with the same grasp head, grid resolution, GPU budget, and hyperparameter search budget as our model. The real-world comparison uses identical physical scenes and object sets. Beating it means higher grasp success at equal view count in packed clutter, with confidence intervals from repeated trials that do not overlap. We expect the gap to be largest at low view counts and under heavy occlusion, where metric anchoring from opaque surroundings should matter most.

The modular RGB-D competitor applies ClearGrasp-style per-view transparency segmentation and depth completion, then fuses the completed depth across views and runs the same grasp network. ClearGrasp reports parallel-jaw grasp success rising from 12% to 72% with its completion, and real-known depth RMSE of 0.039 versus 0.270 for DenseDepth. This baseline isolates whether a hard, single-view mask is worse than a soft, multi-view gate.

Three ablations isolate the key ingredient. The first removes $c(x)$ from the gate's inputs, so the gate must infer reliability from RGB features and view counts alone, as attention in CoBEVT would. The second replaces the gate with unrestricted learned cross-view, cross-modal attention at matched parameter count. The third, an oracle, uses ground-truth transparency masks as the gate, giving an upper bound on what any reliability estimate could deliver.

Why not just…?

Why not just drop the depth camera and use GraspNeRF or MVTrans with more views? This is the most serious threat. MVTrans already reaches 92.9 AUC on the KeyPose benchmark versus 90.0 for stereo KeyPose, and Syn-TODD depth RMSE of 0.124 versus 1.229 for SimNet, without active depth. If RGB-only geometry saturates once there are enough views, our gate adds nothing. The view-count sweep settles this. We report grasp success and transparent-region depth error for both systems at two, three, four, and six views. If our advantage vanishes at the view counts real robots can afford, we will say so, and the contribution shrinks to the low-view regime.

Why not just mask depth with a single-view transparency segmenter? This is the ClearGrasp and TransNet recipe. Its weakness is the cascade: a mask that misses part of a glass edge, or flags a specular highlight on a ceramic mug, passes wrong constraints downstream with full confidence. ClearPose shows accuracy falling from about 58% ADD-S in easy scenarios to 13% under translucent covers, precisely where 2D appearance is ambiguous. The cross-view statistic does not depend on appearance, so the modular baseline above, broken down by ClearPose's heavy-occlusion, translucent-cover, and opaque-distractor scenarios, directly tests whether that independence helps.

Why not just let attention learn reliability end to end? A sufficiently flexible fusion network might discover view inconsistency on its own, which would make our explicit statistic a redundant hand-crafted feature. We take this seriously. The no-statistic ablation and the full-attention ablation exist to measure it. If either matches the gated model on held-out real scenes, the honest conclusion is that the physics prior helps sample efficiency at most, and we will report the gap as a function of training-set size.

Proposed Experiments

The first experiment decides whether the project proceeds. ClearPose sequences come with tracked camera poses from its ProgressLabeller pipeline, so we can compute $c(x)$ on real multi-view RGB-D without any training. We then measure how well it separates ground-truth transparent cells from opaque ones, reported as area under the ROC curve, overall and per adversarial scenario. If the separation is weak on real sensors, for instance because real depth holes dominate rather than reporting background, the gate's inputs are wrong and the design must be rethought before any network is trained.

The second set of experiments measures geometry. We report depth RMSE restricted to transparent pixels on ClearPose and on ClearGrasp's real test images where multi-view sequences exist, comparing against the baselines above. We also report Syn-TODD depth RMSE against MVTrans's reported 0.124, after rendering simulated active depth for that dataset. For category-level pose, we attach a pose head and report ADD-S on ClearPose's heavy-occlusion and translucent-cover splits against the reported scene-wide figures of about 58% and 13%. TransNet's improvement is reported only as 3x over GPV-Pose, with no absolute number found on Lacuna, so we will rerun it.

The third set measures grasping. Simulation follows GraspNeRF's clutter protocol with the depth channel added. Real trials use packed and single-object scenes, on one arm and on a two-arm setup where the second arm's wrist camera supplies views under imperfect relative pose. Two diagnostic checks accompany the grasp numbers. The first is the agreement between the learned gate and ground-truth transparency masks, which tests whether the inspectable intermediate actually means what we claim and lets a failed grasp be traced to a depth-trust error rather than a grasp-head error. The second is a pose-noise sweep showing that pose refinement keeps the gate focused on objects rather than on misregistration.