Ray-Interval Lifting for Open-Vocabulary 3D Perception of Transparent Objects

Robots are increasingly asked to find objects by name ("the wine glass behind the bottle") and reason about them in 3D. The most successful systems for this split the job in two. A large 2D vision-language model decides what each pixel is. A separate geometry network that never sees category labels decides where things are in 3D. This split is what lets them handle object classes they were never trained on. Every such system shares one quiet step, though: it places each pixel's semantic score in 3D at the depth the sensor reports. Glass breaks that step. A depth camera looking at a clear cup returns either nothing or the distance to the table behind it. The system therefore paints "cup" onto the table, and the geometry network, which trusts its inputs, completes a cup-shaped hole in the wrong place.

We propose to repair the lifting step instead of the semantics or the sensor. A semantic score at a glass pixel is spread over the whole interval of the camera ray where the object could physically be, and a category-agnostic 3D network decides where along that interval the object actually sits. The interval is not arbitrary. When the sensor sees through glass to a background surface at distance $d$, the glass must lie strictly in front of $d$. The background reading is a valid upper bound, not noise. Our claim is that keeping this ambiguity until the 3D stage beats committing to a single completed depth beforehand. It should help most where depth completion is known to fail: translucent enclosures, heavy overlap, and regions enclosed by boundaries. It should also keep the zero-shot class generalization that motivated decoupling in the first place.

Core Claim

The decoupled open-vocabulary pipeline of Semantic Abstraction (SemAbs) shows why separation matters. A frozen CLIP model produces relevancy heatmaps, meaning per-pixel scores of how well each pixel matches a text query. These are projected into a voxel grid with sensor depth, and a 3D UNet trained without class identities completes the queried object's occupancy. On novel classes it reached 37.9 IoU where semantics-aware coupled baselines scored 0.0. DCSEG separates geometry and labels in a similar way using Gaussian splatting, a scene representation built from many small 3D ellipsoids. Both assume the geometric evidence is sound. The research file notes that this assumption makes SemAbs project transparent-object relevancy onto the background, and makes photometric splatting produce floaters on refractive surfaces. Neither system has been evaluated on transparent scenes. We found no reported result on Lacuna for open-vocabulary 3D localization or completion of transparent objects on any benchmark.

The claim has two parts. First, the right interface between 2D semantics and 3D geometry on non-Lambertian scenes is a ray interval with physically derived bounds, not a point. Second, the "coupling" that transparent perception papers argue for can be had without coupling to category. MODEST fuses segmentation and depth bidirectionally because untextured glass gives depth networks little to work with. We argue that what depth reasoning actually needs from the 2D side is a material signal ("this pixel's sensor depth is untrustworthy because it is glass"), not a category signal ("this is a wine cup"). A material mask is category-agnostic, so it can feed the geometry network without breaking open-vocabulary transfer.

Research Program

The system has three parts.

The semantic side is unchanged from SemAbs. A frozen vision-language model produces a relevancy heatmap per text query.

A second 2D model produces a class-agnostic transparency mask and confidence. It can be a DINO-feature segmenter in the style of GHOST's TransDINO, which GHOST reports at 92.44% mIoU on ClearGrasp. Its training labels say only "transparent or not." We will check this empirically by holding out categories from its training data too.

The new part is the lifting rule. Opaque pixels with valid depth are lifted as in SemAbs, as a point with free space in front of it. A transparent pixel whose sensor returned depth $d$ contributes its relevancy, weighted by transparency confidence, to every voxel on the ray in the interval $(0, d)$. The voxel at $d$ is marked as genuine background surface rather than as the object. A transparent pixel with no return contributes to the interval up to the working range. The 3D network receives these channels:

  • point-lifted relevancy and occupancy evidence from opaque pixels;
  • ray-interval relevancy from transparent pixels;
  • a free-space or background-bound channel.

Where a learned depth completion is available, it enters as an optional soft channel, a Gaussian bump along the ray at the completed depth. The network can then use completion when it is right and override it when it is not. This makes the method a superset of the obvious cascade, and the experiments ask whether keeping the interval is worth anything beyond that.

The 3D network intersects rays from different pixels, and optionally from a few views. It learns where along each interval a coherent object surface is plausible, using silhouette consistency, support from the table plane, and shape regularities that are generic rather than class-specific.

Training uses synthetic RGB-D scenes rendered with physically simulated glass. Rendering simulates the sensor's failure modes: holes, and see-through returns to the background. Materials, thickness and liquid fill are randomized, as in the material-decoupled data of TransNormal. Supervision is queried-object occupancy, and object categories are split so the test categories never appear in training geometry. The network is trained on relevancy maps, never class labels, which is SemAbs' recipe. That is what should preserve zero-shot behavior.

Baselines

The naive reference point is SemAbs run unchanged on raw sensor depth, with its 3D network retrained on our synthetic data so the comparison isolates lifting rather than training data. We expect it to place transparent objects at background depth. Its score sets the floor and quantifies the background-bleed failure, which so far has only been argued.

The strongest competitor is the cascade: complete the depth first, then run SemAbs on the completed depth. For completion we use the method of Transparent Object Depth Completion. It injects a single-view probability volume into a multi-view cost volume and reports 0.006 m MAE on TransCG, a 40% reduction over FDCT's 0.010 m, plus a 28.2% MAE reduction over TranspareNet in ClearPose's translucent-cover scenario. We also run FDCT as a fast single-view completer. The cascade is the most dangerous baseline because, if completion is accurate, a hard depth estimate is all SemAbs needs.

For fairness, the completers use their released training regime or are retrained on our synthetic set, whichever scores higher on validation. The downstream 3D network is identical in architecture, data and tuning budget to ours. Beating the cascade means higher queried-object IoU on held-out categories, with the gap concentrated in the translucent-cover, heavy-occlusion and liquid scenarios. That concentration is the signature that preserved ambiguity, rather than extra capacity, is doing the work.

The 2D pre-processing competitor is GHOST followed by unchanged SemAbs. GHOST paints opaque texture onto glass so off-the-shelf 3D models can be used; it reports DUSt3R F-score rising from 77.36 to 88.66 and DepthAnythingV3 RMSE falling from 0.023 to 0.018. Here GHOST's output feeds a monocular metric depth model whose output SemAbs then lifts.

The training-free reconstruction competitor is SR3D. It reconstructs a mesh from a single view and anchors it at the table contact edge, reporting 85% simulated grasp success against GraspNeRF's 75%. We voxelize its anchored mesh for the Grounded-SAM-selected object.

The coupled competitor is a MODEST-style model that predicts segmentation and depth jointly, trained on the same data with category labels. It tests whether coupling to category transfers to held-out classes; SemAbs' 0.0 IoU result predicts it will not.

Three ablations isolate the key ingredient:

  • Interval vs. point: replace the ray interval with point lifting at the completed depth, keeping everything else.
  • Bound vs. noise: remove the background upper-bound channel so see-through returns are treated as missing data.
  • Material vs. category mask: swap the class-agnostic transparency mask for a category-specific segmentation, which should help on seen classes and hurt on held-out ones.

Why not just…?

Why not just complete the depth and run SemAbs? This is the real threat. If depth completion were reliable it would make our lifting rule unnecessary. The evidence that it is not reliable everywhere comes from ClearPose: pose accuracy drops from about 58% ADD-S to as low as 13% under translucent covers, and the file notes that enclosed regions have indeterminate absolute depth for completion pipelines. A hard depth estimate that is wrong in exactly these cases sends semantics to a confident wrong place. An interval leaves room for multi-view evidence or support constraints to fix it. Because our network can also consume completed depth, the head-to-head settles this cleanly. If the cascade matches us across all six ClearPose scenarios, the conclusion is that completion plus decoupling already suffices, and we will report it that way.

Why not just make the glass opaque with GHOST first? GHOST models refraction as alpha blending, which its own analysis says fails for thick glass, lenses and filled containers. Those are ClearPose's liquid and translucent-cover scenarios. GHOST is also a per-frame 2D filter with no mechanism for metric scale, and it costs 0.38 s per frame. The GHOST baseline measures whether hallucinated texture produces correct metric placement in 3D, which its reported 2D-side gains do not show.

Why not just fine-tune a coupled model on transparent data? That reintroduces dependence on the training taxonomy. SemAbs' 0.0 versus 37.9 IoU on novel classes is direct evidence of the risk, and TransNet is explicitly limited to six predefined categories. The held-out-category split of the MODEST-style baseline measures this directly.

Why not just reconstruct a mesh and anchor it to the table, as SR3D does? SR3D needs a visible contact edge and relies on a shape prior learned mostly from opaque assets. Clutter, stacking and hollow vessels break it. In isolated tabletop scenes it may well win, and we expect and will report that. The heavy-occlusion and translucent-cover scenarios are where the comparison decides.

Proposed Experiments

No open-vocabulary 3D benchmark with transparent objects exists, so the first deliverable is one built from ClearPose. ClearPose has real RGB-D frames with ground-truth poses and CAD models for 63 objects in six categories. Placing the meshes at their annotated poses gives voxel occupancy for each object, and object names give text queries. Categories are split, for example holding out wine cups and bowls entirely from synthetic training. Results are reported separately for each of ClearPose's adversarial scenarios: new backgrounds, heavy occlusion, translucent covers, opaque distractors, liquids, and non-planar surfaces. Opaque distractors test that the semantic side still separates glass from look-alike opaque objects. TransCG scenes serve as a second real test set, with the depth error of the occupancy surface also reported so we can compare against completion methods on their own metric.

The primary metric is queried-object 3D IoU, following SemAbs' scene-completion protocol. Secondary metrics are:

  • centroid localization error in centimeters;
  • the fraction of an object's relevancy mass that lands on background surfaces, which directly measures bleed;
  • runtime.

Three checks are decisive.

  1. Bleed: naive SemAbs should show large bleed mass and near-zero IoU on transparent objects, and our method should remove most of the bleed. If it does not, the failure mode we target is not the main one.
  2. Cascade gap: our method should beat the cascade mainly in the translucent-cover, heavy-occlusion and liquid scenarios and roughly tie in easy ones. A uniform gain would point to capacity rather than ambiguity. A tie everywhere would refute the claim.
  3. Material-mask swap: a material mask should match a category mask on seen classes and clearly beat it on held-out classes. That would show transparent perception needs material coupling, not category coupling, which addresses MODEST's argument directly.

If these hold, a final small experiment feeds the 3D output to a grasp planner on ClearPose-style real scenes as a sanity check that the improvement matters downstream.