Teaching a Depth Network to See Glass by Touching It

Depth cameras on robots misread glass. Light passes through a clear cup or a glass door, so the sensor either reports the table or wall behind it or returns nothing at all. The usual fix is a neural network that "completes" the missing depth from the color image. These networks are trained offline, on synthetic renders or laboriously annotated real scenes, and they degrade badly when backgrounds, lighting, or clutter change. On the ClearPose benchmark, pose accuracy falls from about 58% to 13% when objects sit behind translucent covers (ClearPose).

A robot arm already carries a sensor that glass cannot fool: its own body. When a fingertip moving slowly along a straight line feels contact, the joint encoders give the exact 3D location of a surface. When it travels a stretch of space and feels nothing, that stretch is certified empty. We propose to turn these gentle "ray probes" into training labels for a transparent-object depth network, in the same way off-road robots turn felt wheel slip into labels for the cameras that saw the terrain beforehand. The goal is a network that, after a few hours of autonomous poking in a new lab, predicts glass depth more accurately without touching anything at deployment time.

Core Claim

Sparse, cheap proprioceptive measurements can correct the specific failure that dominates transparent depth completion: the prediction "falls through" to the background. Two kinds of label do this. A contact gives an exact surface depth. A free-space traverse gives an inequality, saying the surface is not in a given interval. The free-space inequality is the less obvious ingredient. It produces negative evidence, in the sense of "no glass here," without the robot ever colliding with anything. That addresses the positive-label bias that limits proprioceptive self-supervision in navigation, where robots only collect labels for terrain they actually cross (Contrastive Label Disambiguation).

The probe geometry is chosen so the labels avoid refraction. Each probe moves the fingertip along the camera ray of a chosen pixel, so a contact lies on the front surface along an unbent line of sight. The fingertip's 3D position can then be mapped onto the image plane with ordinary pinhole calibration. The research file flags a known weakness of 2D-to-3D feature fusion: refraction breaks the straight-ray projection that methods like PointFusion and 3DMV assume. That weakness bites for anything seen through the glass. It does not bite for points on the near surface, and those are the points we label.

Relation to Prior Work

Our closest prior work uses touch on transparent objects, but only at the moment of grasping. Where Shall I Touch? predicts a stable rim region, pokes it with a GelSight tactile sensor, and combines the measured contact height with the visual mask to plan one grasp. This raises grasp success from 38.9% to 85.2%. Visual-tactile Fusion for Transparent Object Grasping concatenates tactile imprints with image features to correct a grasp pose and classify the object. Both papers motivate this proposal, because they show contact resolves what vision cannot. Both also stop at a single decision. The touch never flows back into the visual model, so the next scene requires touching again. We use touch as a training signal and evaluate the network with touch switched off.

The method side borrows from proprioceptive traversability learning. WayFASTER compares commanded and realized motion to label terrain, then uses odometry to project those labels back onto the camera features that saw the terrain earlier. It assumes opaque surfaces that the depth sensor reads correctly. We reuse its project-the-felt-signal-back logic and move it to the setting where depth sensing itself is the thing being corrected.

Within transparent depth completion, our approach is constrained by two strong vision-only methods.

ClearGrasp shows that surface normals and boundaries are strong cues. It also notes that regions enclosed by boundaries are mathematically indeterminate without extra priors. A single contact inside such a region supplies exactly that missing prior.

Research Program

The system has three parts.

Probing hardware and safety. The hardware is a 7-DoF arm with a wrist force-torque sensor and a rounded, compliant fingertip, viewed by a fixed commodity active-stereo RGB-D camera. No high-resolution tactile skin is needed, since contact detection and joint kinematics suffice. To make a probe, the robot picks a pixel, places the fingertip on that pixel's camera ray at a start depth that it treats as free. The start depth is set in front of the workspace volume, and this is an explicit assumption. The arm then moves outward along the ray under a force threshold low enough to avoid tipping or breaking glassware. The motion stops at contact, or at the depth the raw sensor reports for the background.

Turning probes into losses. Each probe yields a free interval $[d_s, d_e)$ along the ray of pixel $p$, plus a contact depth $d_c$ if contact occurred. Let $\hat d(p)$ be the network's predicted depth at $p$. The probe loss is

$$ \mathcal{L}_{\text{probe}}(p) = \mathbb{1}[\text{contact}]\,|\hat d(p) - d_c| \;+\; \lambda \,\phi\big(\hat d(p);\, d_s, d_e\big), $$

where $\phi$ is a hinge penalty that is zero when $\hat d(p)$ lies outside the free interval and grows linearly with distance inside it. $\lambda$ weights free-space against contact evidence. A probe that reaches the background without contact pushes predictions off the glass and toward the true background. A probe that stops early at glass penalizes the "fell through" prediction twice over: contact pulls the estimate forward, and the background read beyond it is not certified.

The loss is applied to the fingertip's pixel footprint, which spans a few pixels, and is down-weighted at object boundaries. Because each probe labels so little, we keep the opaque-object masking objective of the non-transparent self-supervision paper as a dense regularizer. The probes then correct the transparent-specific bias that the opaque objective cannot see.

Choosing where to probe. Probes cost time, so their placement matters. We compare three strategies under a fixed budget:

  • random pixels inside a transparent-object mask;
  • pixels where an ensemble of depth heads disagrees most;
  • pixels where the predicted depth is close to the raw sensor's background reading, which is the signature of falling through.

The backbone is a lightweight 2D model in the style of FDCT, which runs at 70 FPS, so the adapted network stays usable in closed-loop control. We keep the architecture fixed so that every gain is attributable to the supervision.

Baselines

All baselines share one protocol. They are pretrained on TransCG and then adapted using the same unlabeled target-lab data. That data consists of new objects, backgrounds, and lighting, recorded over the same robot hours. Every method gets the same GPU budget and the same hyperparameter search size.

Naive reference points. The do-nothing reference is the raw sensor depth. The no-adaptation reference is the pretrained network run unchanged in the new lab. Together they show how much of the problem is domain shift and how much is the sensor.

Vision-only self-supervised adaptation. This baseline runs the opaque-object masking self-supervision (TDCNet) on the target-lab frames, with no probes. It is the strongest competitor on the precise question we ask, which is how to adapt without transparent depth labels. Its reported level is about 70% of supervised performance on TransCG. Beating it means a lower transparent-region MAE on target-lab test frames at equal robot time.

Strongest accuracy method. This is the multi-view method of Transparent Object Depth Completion. It reports TransCG MAE of 0.006 m against 0.010 m for FDCT, and a 28.2% MAE reduction over TranspareNet on ClearPose translucent covers. It gets the same number of views the robot camera can actually reach. We report both accuracy and latency. A win for us would be matching or beating its target-lab MAE from a single view at real-time speed. Matching it only on TransCG would not count.

Test-time touch. This baseline feeds probe contacts in as sparse depth inputs at inference, in the spirit of PokePreNet, and gives it the same per-scene probe count. It isolates the question of whether touch must happen during training or merely at deployment.

Ablations. Contact-only, free-space-only, and both are compared, along with the three probe-selection strategies. The free-space-only arm is the key isolation of the proposal's main ingredient.

Why not just…?

Why not just train on more realistic synthetic data with domain randomization? ClearPose documents that state-of-the-art models still drop to 13% ADD-S on real translucent covers, and attributes the gap to scratches, dust, and multi-layer reflections that renderers miss. Simulation may well narrow the gap. Our protocol tests that by including a synthetic-pretrained starting point. What simulation cannot provide is labels from the specific deployment room, and that is the kind of shift the target-lab test measures.

Why not just use multi-view reconstruction (NeRF, Gaussian splats, or stereo cost volumes) and skip contact? Multi-view matching locks onto the background visible through glass. Transparent Object Depth Completion needs single-view anchors precisely to avoid this, and it is computationally heavy. Clear-Splatting and Residual-NeRF need wide viewpoint coverage per scene, which a fixed camera or a single reach rarely provides. If the multi-view baseline, given reachable views, matches our single-view network, our accuracy advantage disappears. We would then report the result as a latency and viewpoint trade-off rather than an accuracy gain.

Why not just touch at test time, as PokePreNet does? This is the real threat. For a single tabletop grasp of a rimmed glass, test-time poking may be as good as anything we train. We settle the question two ways.

  • First, we evaluate depth and grasping with touch disabled. This is the regime that matters when probing is slow, when objects are densely packed so a poke can topple neighbors, or when the target has no pokeable rim.
  • Second, we count total contacts per successful grasp over a long session. If our trained network does not reduce contacts per grasp relative to always-poke, the training contribution is not worth its cost.

Why not just ask a vision-language model? Physically Grounded Vision-Language Models shows that base models reach only 66.2% on physical-concept questions. Even the fine-tuned model reasons about categories such as "fragile" and "transparent," not metric depth. So it complements this work and does not replace it.

Proposed Experiments

Data collection. We collect a target-lab dataset with transparent objects from the ClearPose and TransCG categories plus new glassware, arranged on new backgrounds and including scenes with translucent bins. For evaluation only, ground-truth depth comes from fitting CAD models to object poses, following the annotation approach of ClearPose. No evaluation label is used in training.

Primary metrics. The primary metrics are MAE and RMSE on transparent pixels and the rate of predictions falling through to the background. Reference points from the file are 0.006 m MAE on TransCG (Zhou et al.) and 0.039 RMSE on the ClearGrasp real-known set (ClearGrasp). We also re-evaluate on TransCG and on ClearPose translucent-cover frames to check that adaptation does not erase source-domain performance.

Scaling experiment. We plot target-lab MAE against probe count, from 0 up to a few thousand probes, and against an equal number of human-annotated frames. This tells practitioners what one robot-hour of probing is worth.

Downstream grasping. We run parallel-jaw grasping on the target lab with touch disabled. The references are ClearGrasp's 72% parallel-jaw success and PokePreNet's 85.2% with touch. We also log breakages and tip-overs during probing, so the safety cost is reported rather than assumed away.

Decisive checks. The proposal succeeds if four things hold:

  • the full method beats vision-only TDCNet adaptation on target-lab transparent MAE at equal robot time;
  • free-space-only probing alone measurably reduces the fall-through rate, showing that negative evidence is doing work;
  • touch-free grasping approaches test-time-poke success while using fewer contacts per grasp;
  • uncertainty-directed probing outperforms random probing at small budgets.

The proposal fails if these gains vanish once the vision-only regularizer is tuned equally hard, or if probe labels at fingertip resolution cannot move predictions beyond a few pixels around each contact.