Do Transparent-Object Depth Benchmarks Hide the Value of Sensor Realism?

Depth cameras fail on glass. The infrared pattern they project either passes through the object and returns the distance of the table behind it, or bounces off at an angle the camera never sees. Either way the raw depth image shows holes, background readings and scattered stray points where the object should be. A family of "depth completion" networks repairs these maps, and they are ranked almost entirely by averaged per-pixel errors such as root-mean-square error (RMSE) and mean absolute error. This proposal argues that those averages are blind to the errors that matter most on transparent objects. A few dozen leaked background pixels, a cloud of flying points at the rim, or a flattened bowl interior barely move an average over thousands of pixels, yet they can wreck a grasp or a pose fit.

These are exactly the errors that depend on how a specific sensor fails. If so, the field may have been unable to see whether realistic sensor-noise modeling helps transparent-object perception, because its metrics cannot register the difference. We propose two things: a shape-aware, artifact-aware evaluation calibrated against downstream pose accuracy, grasp quality and human judgment, and a controlled test of whether sensor-faithful synthetic corruption improves synthetic-to-real transfer once it is measured properly.

Core Claim

The claim has two parts, and both can fail. First, at matched RMSE on transparent objects, downstream pose and grasp outcomes vary widely. The variation is driven by two error types: residual sensor artifacts, meaning raw readings the network copies through instead of correcting, and distorted surface orientation. An evaluation that renders the completed geometry as a matte shape and separately counts 3D artifact points will rank published methods in the same order as downstream outcomes far better than RMSE does.

Second, training on synthetic data corrupted with a sensor-specific failure model will improve that evaluation clearly more than RMSE, compared with random holes or morphological erosion. That pattern would mean the benefit of sensor realism has been partly invisible under current leaderboards. If the gains show up equally under RMSE, the "hidden value" hypothesis is falsified, and we report it as such.

One conceptual point lets us reuse existing perceptual metrics. The target of transparent depth completion is the object's shape: ClearPose's ground truth comes from aligned CAD models. Shading that shape as if it were matte is therefore the right way to judge it, even though the object is optically glass. The Lambertian assumption criticized in prior perceptual losses is a limitation for modeling appearance, not for evaluating recovered geometry.

Grounding in Prior Work

Two papers motivate the metric side. Perceptual Deep Depth Super-Resolution renders depth maps under three orthogonal lights with a diffuse shading model and shows that RMSE can invert perceived quality: 46 mm RMSE looked bumpy while 99 mm looked correct. Its subjects preferred visually optimized outputs over 85% of the time. DR-KFS replaces Chamfer distance with multi-view rendered keypoint matching on ShapeNet and was preferred by humans 71% of the time. Both studied opaque objects, and neither tied its metric to a robot task. We reuse the first paper's shading construction, apply it to transparent-object geometry, and validate it against task outcomes as well as people.

Two further papers show the downstream gap from the transparent side. ClearGrasp found surface normals more predictive of grasping than depth RMSE. ClearPose reports pose accuracy of roughly 58% ADD-S in easy scenes falling to 13% under translucent covers. Neither paper builds an evaluation that would catch this.

The sensor side is constrained by existing simulators. Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation and Mesh2Point / QBST show that simulating the real acquisition pipeline beats domain randomization (0.648 vs 0.584 IoU) and beats clean training (64.0% vs 46.7% on DepthScanNet10). Both handle only opaque or metallic surfaces, and the former names transparent materials as unsolved. Full refractive simulation of projected infrared patterns is the obvious next step and is expensive. We do not build it; we take a cheaper data-driven route, described below.

Self-Supervised Learning for Transparent Object Depth Completion supplies our main corruption baseline. It simulates glass failures by eroding opaque-object masks and reaches about 70% of supervised performance. AdaptPoint supplies the learned-augmentation alternative, which perturbs and masks coordinates without regard to optics.

Research Program

The work has three components.

Error taxonomy and controlled perturbations. We start from ground-truth depth of transparent objects in ClearPose and TransCG. To that ground truth we apply parametric error families observed in real completion outputs:

  • smooth global offset;
  • high-frequency ripple;
  • interior flattening;
  • boundary bleed;
  • copied-through background readings from the raw sensor map;
  • flying points at the rim.

Each family is scaled to hit a set of fixed RMSE levels. For every perturbed map we measure three outcomes:

  • ADD-S from a fixed pose estimator run on the completed depth;
  • an analytic grasp-quality score from a fixed grasp planner, compared against grasps on the CAD mesh;
  • human pairwise preference on matte renderings.

This isolates the misalignment directly. If outcomes are flat at fixed RMSE, the premise fails.

A sensor-aware shape metric. On the object mask $\Omega$, we define

$$M = \frac{1}{|\Omega|}\sum_{p\in\Omega}\frac{1}{3}\sum_{k=1}^{3}\big(\max(0,\hat n_p\cdot l_k)-\max(0,n_p\cdot l_k)\big)^2 \;+\; \lambda\,A_\tau .$$

Here $\hat n_p$ and $n_p$ are predicted and ground-truth surface normals at pixel $p$, computed from depth by finite differences. The $l_k$ are three orthogonal light directions, following the perceptual super-resolution paper, and the clamp keeps back-facing pixels unlit as in a matte render. $A_\tau$ is the fraction of back-projected predicted object points lying farther than $\tau$ from the ground-truth surface; this term catches leaked and flying points. We fit only $\lambda$ and $\tau$, on the perturbation study, and freeze them before touching any real method outputs. The point of the metric is that it is small enough to be audited, not that it is learned.

A data-driven sensor failure emulator. TransCG and ClearPose pair raw sensor depth with ground truth on real glass. From these pairs we train a small network that maps geometric context to a per-pixel distribution over four outcomes: valid at the surface, valid at the background, missing, or outlier. The geometric context consists of the angle between normal and viewing ray, distance to the silhouette, the depth gap to the background, and object thickness rendered from CAD. We train it per sensor. The emulator is then applied to the synthetic ClearGrasp renderings to produce corrupted inputs that mimic how a particular camera fails. It is a statistical stand-in for physical refraction simulation: cheaper, sensor-specific by construction, and limited to failure modes present in the paired data. Its realism is checked by a held-out likelihood of real raw maps and by how well a classifier can tell emulated corruption from real corruption.

Baselines

Raw-input reference. The naive point is to apply no completion at all and feed raw sensor depth to the pose estimator and grasp planner. This sets the floor for every downstream number.

Strongest completion network on current metrics. This is the hybrid single-view and multi-view method of Transparent Object Depth Completion. It reports a TransCG MAE of 0.006 m against FDCT's 0.010 m, and a 28.2% MAE reduction over TranspareNet on ClearPose's translucent-cover scenario. Alongside it we run FDCT and TODE-Trans, whose code is public. These matter because they define the current ranking we expect to be reordered. Where a released checkpoint does not exist, we retrain on the same splits with the authors' settings, and every method feeds the identical frozen pose estimator and grasp planner.

ClearGrasp and GHOST. ClearGrasp reports 0.039 RMSE on Real-Known. GHOST with DepthAnythingV3 reports 0.018 RMSE on ClearGrasp, against 0.023 without opaquification. Both are included because they optimize normals or appearance rather than raw depth, and so should fare relatively better under $M$ than under RMSE if our claim holds.

Metric baselines. We compare $M$ against RMSE, MAE, REL, threshold accuracy, mean normal angular error, the shading term alone, and a rendered-view keypoint score in the spirit of DR-KFS. Each is judged by Kendall rank correlation with ADD-S and grasp quality across all method outputs. Beating the baselines means a statistically higher rank correlation than both RMSE and normal angular error on held-out ClearPose scenes.

Corruption ablations. All corruption variants use one fixed completion architecture, FDCT for speed, with identical synthetic data, epochs and tuning budget, so that the corruption model is the only change. The variants are:

  • clean synthetic input;
  • random holes with Gaussian jitter;
  • the erosion masks of the self-supervised paper;
  • AdaptPoint-style learned masks;
  • the emulator;
  • the emulator trained on a different sensor, to test whether sensor specificity, rather than realism in general, is what matters.

Why not just…?

Why not just use surface-normal angular error, which ClearGrasp already showed predicts grasping? This is the most serious threat. Normal error may capture most of what the shading term does, since the shading term is a smooth function of normals. However, neither normal error nor shading sees the leaked background points and rim outliers that $A_\tau$ targets. The perturbation study tests this directly: if normal error ranks methods as well as $M$, we drop the shading term and report that a known metric was simply underused.

Why not just train on real TransCG labels and skip synthetic corruption? For one sensor and one environment that is the right baseline, and we include it. The question here is transfer to scenes and sensors without labels. The test is to train on synthetic data with emulated corruption and evaluate on ClearPose's adversarial scenarios, compared against a TransCG-trained model evaluated on the same scenes.

Why not build a full physical simulator of infrared patterns refracting through glass? This would be more faithful, as the structured-light and Mesh2Point results suggest for opaque scenes. It is also costly and not differentiable, and it requires refractive indices and thickness we rarely know. The emulator is a cheaper test of whether sensor-specific failure statistics matter at all. If they do, the case for building the physical simulator becomes concrete.

Why not opaquify with GHOST and use a foundation model? GHOST's alpha-blending assumption is stated to break on thick glass. More to the point, it is evaluated by RMSE and F-score, so whether it removes artifact points is unknown. We measure that under $M$ and $A_\tau$.

Proposed Experiments and Decisive Checks

The decisive experiments run in order of risk.

  1. Matched-RMSE perturbation study on ClearPose and TransCG ground truth. Success means downstream ADD-S and grasp quality vary substantially within a fixed RMSE level, and the variation is explained by artifact fraction and orientation error.
  2. Re-ranking of public method outputs. Kendall correlation with downstream outcomes is computed for each metric, and we report which leaderboard orderings flip.
  3. Corruption ablation with a pre-registered prediction. The emulator's advantage over erosion and random holes should be at least twice as large, in relative terms, under $M$ and ADD-S as under RMSE, when training on ClearGrasp synthetic data and testing on ClearPose, especially the translucent-cover and liquid scenarios.
  4. Training with $M$ as an auxiliary loss on top of the best corruption model. We check that it improves downstream outcomes and does not merely improve the metric it optimizes; the guard against overfitting the metric is that pose and grasp are measured by separate frozen systems.

Throughout, ClearPose's 13% ADD-S under translucent covers is the regime where we expect the largest gap between what current metrics report and what a robot experiences.