Depth cameras give wrong readings on glass. Where a drinking glass stands, the camera usually reports the table behind it, reports nothing, or reports a few scattered stray points. Almost every transparent-object method treats these pixels as holes to fill in. They mask the bad depth and use the color image to guess a corrected depth map. We propose the opposite. For a given camera, the bad reading is mostly a predictable result of where the glass is: if the camera returned the background at a pixel, light passed through something there; if it returned nothing, light bounced away at a steep angle. We want to build a 3D voxel grid of the scene (a lattice of small cubes, each labeled filled or empty) that stores two quantities per cube: whether it is physically solid, so a robot can avoid colliding with it, and how much light passes through it. The grid is trained by predicting what this specific sensor would actually read, failures included.
The stakes are practical. Robots in kitchens and laboratories have to grasp, pour into, and avoid hollow glassware, and these tasks need the back wall, the inside of the container, and the object hidden behind another clear vessel. A single corrected depth map stores one depth per pixel and cannot hold any of that. On ClearPose, the best methods reach about 58% pose accuracy (ADD-S) on easy scenes and drop to 13% when objects sit inside translucent covers. That is the multi-layer regime where one depth per pixel breaks down.
Core Claim
Existing voxel occupancy models learn geometry in part by rendering depth from the grid and matching it to sensor depth. GroundingOcc shows that this rendered-depth supervision adds roughly 2.5% accuracy on language-guided occupancy grounding. The rendering assumes each camera ray stops at the first occupied voxel. On glass, that assumption is false, and the loss pushes the grid toward the wrong answer: it will either erase the glass or move it onto the background.
Our claim is that replacing first-hit rendering with a sensor-specific observation model has two effects. The model is a small per-camera function that, given the solid and transmissive voxels a ray passes through, predicts a distribution over three outcomes: front-surface return, pass-through to whatever lies behind, or dropout. First, it removes these erroneous gradients. Second, it turns the corrupted transparent pixels in raw, unlabeled RGB-D video into supervision for the 3D grid.
The new element is the observation model. Neither the voxel grid nor the sensor simulation is new.
The model rests on one physical simplification, which we state openly. For thin-walled hollow glassware such as beakers, bottles, and cups, the sideways shift of a ray crossing two nearly parallel wall surfaces is small. A straight ray with a transmission probability is therefore a reasonable approximation. Solid glass objects bend rays strongly and violate it. This is the reverse of the regime covered by refraction-tracing reconstruction. Differentiable Refraction-Tracing and NeTO model refraction exactly for solid objects from 4 to 72 calibrated views. NeTO explicitly discards rays that refract more than twice, and every ray through a hollow vessel refracts at least four times.
Research Program
The work has three parts.
Physical sensor simulator. We extend the physically based structured-light simulator of Bai et al. to transmissive materials. That simulator projects virtual Gray-code patterns (sequences of stripe images used to match projector and camera pixels), ray-traces them, and runs the real decoding algorithm. On opaque metal and plastic parts it reproduces real artifacts such as edge dropouts and floating points. Its authors name transparent surfaces as future work. We add dielectric glass shaders with Fresnel reflection, the angle-dependent split of light into reflected and transmitted parts. We also add hollow-vessel geometry and liquid fills, and run the same decoding pipeline. This is the sensor-specific noise component.
This simulator is expensive and does not need to be differentiable. We use it offline for two purposes: to generate synthetic training scenes, and to fit the small observation model.
Observation model. The fitted model predicts outcome probabilities from four inputs:
- incidence angle at the first transmissive voxel,
- accumulated transmittance along the ray,
- depth of the next solid voxel,
- the camera's projector–camera baseline.
The model is fitted separately per sensor, and it is cheap and differentiable.
Occupancy network and loss. A voxel network takes one or a few RGB-D frames and outputs two channels per voxel: collision occupancy and transmittance. We reuse height-preserving voxel feature aggregation in the spirit of SliceSemOcc, because thin vertical glass walls are exactly the features that pooling over height destroys. The network is trained with two losses:
- a voxel loss on synthetic ground truth;
- a loss defined as the likelihood of the raw sensor depth under the observation model. This applies to both synthetic and real unlabeled frames.
The dataset analysis then checks where the gains come from. It splits errors into three groups: front surfaces, back walls and interiors, and objects behind other transparent objects.
Baselines
Raw-depth reference. The naive reference fuses raw sensor depth directly into a voxel grid, with no correction. This measures how much glass is simply absent.
Strongest competitor: corrected depth fused into voxels. We take Transparent Object Depth Completion (TODC) and fuse its corrected depth maps across the same frames into the same voxel resolution. TODC builds an internal 3D probability volume and a multi-view cost volume, but outputs a corrected 2.5D depth map. It reports MAE of 0.006 m on TransCG, down from FDCT's 0.010 m, and a 28.2% MAE reduction over TranspareNet on ClearPose translucent covers. It already uses 3D structure and multiple views, so it is the most direct threat.
We also include FDCT fused the same way as a fast option at 70 FPS. All completion methods receive the same frames, training data including our simulated scenes, and tuning budget.
FetchBot-style foundation depth. Following FetchBot, we replace sensor depth with DepthAnything predictions and add an occupancy head on top. FetchBot reports 89.95% real fetching success in clutter. It avoids sensor noise entirely rather than modeling it.
Two ablations that isolate the key ingredient. Both use our exact network:
- First-hit ablation: the depth loss uses GroundingOcc-style first-hit rendering.
- Masking ablation: the depth loss simply ignores pixels on transparent objects.
A third, weaker ablation replaces the physically fitted observation model with one whose parameters are domain-randomized.
What counts as beating the competitors. We should beat fused TODC on back-wall and interior voxel IoU and on behind-glass voxel IoU in ClearPose's translucent-cover and heavy-occlusion scenes, with confidence intervals over scenes. We should also beat the masking ablation by a clear margin. We do not expect to beat TODC on front-surface depth MAE, and will report it either way.
Why not just…?
Why not just complete the depth in 2.5D and fuse it into voxels? This is the real threat. With enough viewpoints, fusing corrected depth maps could recover back walls seen from the other side. Its weakness is that each corrected map still holds one depth per pixel. TODC itself documents depth smearing where transparent objects overlap. The single-frame and few-frame experiments, and the comparison stratified by number of views, decide this directly. If fused TODC matches us at around five views, our advantage is limited to few-view settings, and we will say so.
Why not just mask transparent pixels and let the color image do the work? This is what most methods effectively do, and it may be enough. The masking ablation tests it with everything else held fixed. If masking matches the full observation model, our central claim is false: the corrupted depth carries no usable signal.
Why not just add random holes and noise instead of simulating the sensor? Bai et al. report that on opaque parts, randomized depth reached real detection IoU 0.584 versus 0.648 for physical simulation. Quasi-Balanced Self-Training argues that real depth errors are structured rather than random. Neither result has been shown on glass, so our domain-randomized observation-model ablation is the actual test.
Why not just use monocular foundation depth? ClearGrasp reports monocular DenseDepth at RMSE 0.270 versus 0.039 for its sensor-guided method on Real-Known objects. TODC describes image-only estimators being pulled toward the high-contrast background behind glass. DepthAnything is newer than both, so the FetchBot-style baseline is run rather than dismissed.
Why not just fit a radiance field? The refraction-tracing methods need calibrated multi-view capture and minutes of optimization per object. They assume solid objects, so they do not fit a robot looking at hollow glassware once.
Proposed Experiments
Data. The primary benchmark is ClearPose. Its CAD models and per-frame poses let us voxelize ground-truth occupancy, including back walls and interiors, without altering the scenes. Its six adversarial scenarios give stratified test splits: translucent covers, heavy occlusion, and liquids are the decisive ones. TransCG and the ClearGrasp real test set provide front-surface depth MAE and RMSE, computed by rendering our grid. These numbers can be compared against the reported 0.006 m (TODC on TransCG) and 0.039 RMSE (ClearGrasp, Real-Known).
Simulator check. Before any network is trained, we compare simulated and real raw depth on matched glass objects. Comparison uses per-pixel outcome histograms: return, pass-through, and dropout, as functions of incidence angle. If the extended simulator does not reproduce the real dropout pattern, the observation model has no basis and we stop.
Decisive checks. The main checks are back-wall IoU and behind-glass IoU against fused TODC as the number of input views goes from one to five. Alongside them we run the masking ablation, which tests whether corrupted depth is informative at all.
Training on real unlabeled data. We train with and without the likelihood loss on real unlabeled ClearPose video. This tests whether the loss lets real data improve the grid without labels.
Known limitation, measured. We evaluate on the solid objects in the DRT dataset, where we expect the straight-ray assumption to fail, to measure how large that failure is.
Downstream robot test. A small real-robot study measures collision-free reaching and grasping among clustered glassware, comparing our grid with fused TODC in the same planner.
References
- ClearPose: Large-scale Transparent Object Dataset and Benchmark
- A Coarse-to-Fine Approach to Multi-Modality 3D Occupancy Grounding
- Differentiable refraction-tracing for mesh reconstruction of transparent objects
- NeTO: Neural Reconstruction of Transparent Objects with Self-Occlusion Aware Refraction-Tracing
- Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation
- SliceSemOcc: Vertical Slice Based Multimodal 3D Semantic Occupancy Representation
- Transparent Object Depth Completion
- FDCT: Fast Depth Completion for Transparent Objects
- FetchBot: Learning Generalizable Object Fetching in Cluttered Scenes via Zero-Shot Sim2Real
- Quasi-Balanced Self-Training on Noise-Aware Synthesis of Object Point Clouds for Closing Domain Gap
- Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation