Sampling Where Glass Is Curved and the Depth Camera Lies

Depth cameras fail on glass in a structured way. Infrared light passes through the object and returns the depth of the table behind it, which produces "phantom" depths. Light glancing off a curved rim scatters away from the receiver, which leaves holes. Methods that repair these depth maps spend their effort evenly. They march rays through a uniform voxel grid (RGB-D Local Implicit Function for Depth Completion of Transparent Objects) or voxelize a point cloud uniformly (Seeing Glass: Joint Point-Cloud and Depth Completion for Transparent Objects). As a result, flat glass sides get as many queries as the thin rims and stems where grasps actually succeed or fail. We propose to train and run a transparent-object depth completer with a fixed query budget. The budget is allocated in proportion to two predicted quantities: how sharply the surface bends, and how likely this particular sensor's reading is wrong at that spot.

The bet is that these two signals are complementary, and that neither alone is enough. Curvature alone wastes queries on well-measured bends, such as the opaque table edge. Sensor unreliability alone treats every hole the same and misses phantom points that look valid but sit centimetres off the true surface. Their product targets the region that existing methods smooth over: the curved boundary where the camera's failure is systematic. The idea is cheap to test and has a clean way of being falsified, which is part of why it is worth doing.

Core Claim

At equal query budget, importance sampling an implicit depth-completion network by a predicted curvature × predicted-sensor-error density will reduce error on high-curvature regions of transparent objects more than any of the following: uniform sampling, sampling by invalid-depth mask alone, sampling by curvature alone, or spending the same compute on denser uniform sampling. Whole-image error should not get worse.

Two design choices separate this from naive transplants of curvature sampling. First, curvature is not computed from the corrupted depth. The research file's Gap 1 notes two weaknesses there. Graph high-pass filters (Fast Resampling of Three-Dimensional Point Clouds via Graphs) act as spatial differentiators and would amplify refractive scatter. Second derivatives of a signed distance field are unstable even on clean LiDAR, enough that CLAP needs a uniform warm-up period. We instead take curvature from RGB-predicted surface normals. These are the same kind of normals that Clear Grasp, TransNet, and GHOST already show can be estimated on glass.

Second, the sensor model is learned from real paired data, not simulated. It follows the two-regime split from PRISM: "no signal" holes versus "bad signal" wobble. Real datasets with raw and ground-truth depth let us supervise per-pixel error directly. This sidesteps the unsolved problem of simulating refraction in active sensors, which Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation explicitly leaves open and which Mesh2Point does not model.

Research Program

The base system is a local implicit depth function in the style of LIDF. That model casts camera rays through voxels and asks a small network, conditioned on local RGB and depth features, where each ray hits the surface. LIDF reports a mean absolute error (MAE) of 0.020 m on real unseen objects at 0.09 s per image. We add two lightweight heads to its image encoder.

The first head predicts a unit normal field $\hat{n}$. From it we compute a curvature proxy $\hat{c} = \tfrac{1}{2}|\nabla\cdot\hat{n}|$. This is half the divergence of the normal field, which is the standard expression for mean curvature. It is evaluated with finite differences on the image grid.

The second head predicts, per pixel, two probabilities. $\hat{p}_{\text{hole}}$ is the probability that the sensor returns no depth. $\hat{p}_{\text{bad}}$ is the probability that the returned depth is off by more than a threshold $\tau$. Both heads are supervised on real datasets that pair raw sensor depth with ground truth: TransCG, TODD, and the ClearGrasp real set with its spray-painted ground truth. These labels describe the specific camera that captured each dataset, so the model is "sensor-specific" by construction.

Ray queries are then drawn from a mixture density over pixels $u$:

$$q(u) \propto \lambda_0 + \hat{c}(u)^{\alpha}\,\big(\hat{p}_{\text{hole}}(u) + \hat{p}_{\text{bad}}(u)\big)$$

Here $\lambda_0$ is a uniform floor that keeps every pixel reachable and removes the need for a warm-up period. The exponent $\alpha$ controls how sharply queries concentrate on curvature. Along each sampled ray, depth samples are also concentrated in a band around the raw reading, with the band widened when $\hat{p}_{\text{bad}}$ is high. The same density decides where the training loss is evaluated and where inference-time refinement queries go, under a fixed total number of queries per image. A final step interpolates the sparse implicit predictions back to a dense map, guided by the normals.

The analysis has three parts. We measure how strongly ground-truth curvature and ground-truth sensor error co-occur on each dataset. If they are nearly identical, the product reduces to one term, and we will report that. We map where the reliability head is wrong, especially for liquid-filled vessels in TODD. And we sweep the query budget to find where adaptive sampling stops mattering.

Baselines

The naive reference is the raw sensor depth, together with the same implicit network using uniform ray sampling at our query budget. The second of these is our LIDF re-implementation, so the only change in our method is where queries go.

The strongest published competitors on ClearGrasp real data are LIDF and TranspareNet. LIDF reports a real-world unseen MAE of 0.020 m. TranspareNet reports an RMSE of 0.032 on the ClearGrasp Unseen split, against 0.044 for ClearGrasp on that split. ClearGrasp itself reports RMSE of 0.039 on known and 0.028 on novel objects. These numbers come from those papers' own protocols. We will rerun LIDF on identical splits, preprocessing, and hardware, and quote the published figures only for orientation.

On TransCG, the speed-oriented baseline is FDCT, which reports more than a 16% improvement over DFNet at 70 FPS. No reported absolute TransCG error for FDCT was found on Lacuna, so we will measure it. TODE-Trans reports relative gains of 25% RMSE and 21% REL over convolutional baselines; no reported absolute number was found on Lacuna for it either. Both will be retrained on the same TransCG training split under the same tuning budget, which is a fixed number of hyperparameter trials per method.

The ablations isolate the product term. They are: sampling by $\hat{p}_{\text{hole}}$ only; by $\hat{p}_{\text{hole}}+\hat{p}_{\text{bad}}$ only; by $\hat{c}$ only; by an additive rather than multiplicative combination; and by curvature computed from the second derivative of a field fit to the raw depth, which tests the Gap 1 claim directly. Two oracles give an upper bound: one uses ground-truth curvature and one uses ground-truth error maps.

Beating the field means two things. On the high-curvature evaluation band, our error must be lower than uniform LIDF and every single-signal ablation, with confidence intervals across three seeds that do not overlap. And whole-image RMSE must be no worse than uniform LIDF.

Why not just…?

Why not just sample where the depth is missing? This is the most serious threat. Holes on glass cluster at rims precisely because glancing angles there scatter the infrared light, so an invalid-depth mask may already be a good curvature detector. The mask-only and mask-plus-bad-signal ablations settle it. If mask-only matches the full method on the curvature band, we drop the curvature term and report that sensor failure is a sufficient sampling signal. Our prior that it will fall short comes from TranspareNet's finding that refracted, non-missing returns still carry information but are displaced. A mask treats those pixels as trustworthy and puts no queries there.

Why not just add Gaussian or random-dropout noise and keep uniform sampling? On opaque objects this already loses badly to structured noise models: 46.7% versus 64.0% DepthScanNet accuracy in Quasi-Balanced Self-Training on Noise-Aware Synthesis of Object Point Clouds for Closing Domain Gap, and 38.9% versus 92.1% policy success in PRISM. More importantly, augmentation changes the training data, not where the model spends its capacity. It is therefore orthogonal to our proposal and can be combined with it. We include it as an extra arm on uniform LIDF.

Why not just spend the budget on denser uniform sampling or a finer voxel grid? If doubling uniform queries closes the gap, adaptive sampling is not worth its complexity. We run uniform sampling at 2× and 4× our budget and report the budget at which uniform sampling matches us. The method is only useful if that multiplier is well above 1.

Why not just weight the loss by curvature instead of resampling? A curvature-weighted loss on uniformly sampled queries changes the emphasis of learning but leaves inference-time resolution unchanged. It is a cheap alternative and gets its own ablation. If it matches resampling at training time, we keep resampling only at inference.

Proposed Experiments

Training uses the TransCG training split. Evaluation covers TransCG test, the ClearGrasp real known and novel splits, and TODD, which tests liquid-filled vessels where the reliability head is expected to struggle. Metrics are RMSE, MAE, REL, and the fraction of pixels within 1.05, 1.10, and 1.25 of ground truth, computed over transparent-object masks. We also report the same metrics restricted to a band of the top 20% of ground-truth curvature pixels on each object, plus mean normal error in degrees, which ClearGrasp reports as 18.72° and 18.09°. Latency is reported per image on one GPU.

The first decisive check is the curvature-band comparison against the mask-only ablation, described above. The second is cross-dataset transfer. We train the reliability head on TransCG and test on ClearGrasp real and TODD without retraining it. These were captured with different setups, so a large drop would show that the sensor model is genuinely sensor-specific and needs per-camera calibration data. The third is an oracle-gap analysis. If ground-truth error maps help far more than predicted ones, the bottleneck is the noise model rather than the sampler, and future effort should go to physical simulation of refraction.

If budget allows, a small grasping study on rims and stemware would follow ClearGrasp's protocol, which reports parallel-jaw success rising from 12% to 72%. It would test whether gains on the curvature band translate into better grasps.