Depth cameras largely fail on glass and clear plastic. Light passes through the object, so the sensor either returns nothing or returns the depth of the table behind it. Networks that fill in these holes, a task called depth completion, need training images with the correct depth, and those labels are expensive to capture for transparent objects. A recent workaround sidesteps labels altogether. It deletes the depth inside ordinary opaque objects, whose depth the sensor measures correctly, and trains a network to restore it (Self-Supervised Learning for Transparent Object Depth Completion Using Depth from Non-Transparent Objects). That trick reaches roughly 70% of fully supervised performance on real transparent objects. Its simulated holes are simple, though. Each one is a single whole-object mask with a uniform border kept intact. The opaque RGB image inside the hole still shows the object's own texture.
Real transparent objects break both of those simplifications. A lidded jar, a goblet, or a flask inside a translucent bin has internal depth steps between its parts. Its interior pixels also show the background texture that passes through the glass rather than any surface of the object. This proposal makes the simulated holes look more like real transparent objects in two ways. First, each hole is organized by the object's parts, so the network has to reconstruct internal depth steps. Second, background texture is composited into the hole, so the network has to learn to ignore what shows through. We then add attention gates to the network's skip connections, the shortcuts that carry high-resolution early features straight to the output layers. The gates decide which fine edges inside the hole are genuine part seams and which are background leaking through.
Core Claim
Masking-based self-supervision for transparent depth completion is limited by the realism of its simulated holes, not only by network design. Two things are missing from current holes, and both are free to generate from opaque RGB-D data. One is part structure, meaning internal depth discontinuities. The other is see-through appearance, meaning background texture visible inside the hole. Once both are present in training, it becomes useful to gate skip connections with a signal that knows where the part seams are. Without seam-aware holes, a gate has no reason to keep interior edges. Without see-through compositing, it has no background bleed to learn to suppress.
The prediction is specific and testable. The gains should concentrate at internal depth discontinuities and in nested-transparency scenes such as ClearPose's translucent covers. Aggregate error over whole transparent regions should improve less.
Research Program
Part-structured holes. Each training frame comes from a real RGB-D scene that contains opaque objects, as in TransCG. We run the Segment Anything Model (SAM), which proposes masks at several granularities for a single prompt, as described in Flaws can be Applause. This gives nested whole-level and part-level masks for each opaque object, and we delete depth at the whole level.
We treat a SAM part boundary as a real seam only where the measured opaque depth actually jumps across it. This matters because the part-whole literature documents how unreliable granularity choices are. SAM defaults to wholes (Guided SAM), and slot-based methods flip between parts and wholes (Adaptive Slot Attention). We therefore let geometry, not the segmenter, decide which part boundaries count. The verified seams become a free auxiliary target: a seam map inside the hole. The erosion that preserves the object border in the original method is kept at the outer contour only.
See-through compositing. Inside each hole, the RGB image is replaced by a blend of the object's own pixels and a displaced crop of the surrounding background:
$$\tilde I(u) = (1-\beta)\, I(u) + \beta\, I(u+\delta(u)), \quad u \in M.$$
Here $I$ is the original image, $M$ is the deleted-depth mask, $\beta \in [0.5, 0.9]$ is a randomly sampled transmission strength, and $\delta(u)$ is a smooth random displacement field that mimics refraction. The depth target stays the opaque object's true surface. The network therefore sees exactly the conflict that causes background bleed on real glass: background edges appear in the image, but the answer is the foreground surface. This is a crude physical model. We are not claiming realism, only that the network learns to distrust transmitted texture.
Seam-aware gated skips. We build on a single-view U-Net completion network, since FDCT shows such networks can be fast and accurate (FDCT). Its plain skip connections are replaced with additive attention gates in the style of Attention Gated Networks. In such a gate, a coarse decoder feature produces a per-pixel weight between 0 and 1 that scales the encoder feature before it is passed on.
We change the gating input. The decoder feature is concatenated with the deficit mask and with the decoder's own current seam prediction. Inside the hole, the gate can then pass encoder edges that coincide with predicted seams and suppress the rest.
Constraint and risk. A cautionary result shapes this design. In self-supervised driving, gates learned to discard reflections as noise (Self-Supervised Attention Learning for Depth and Ego-motion Estimation). On glass, that behavior would also discard the faint cues that mark the object's true boundary. We will log gate values along ground-truth transparent contours and seams on real test data. If the gates are near zero there, the design has failed in the way that paper warns about.
Baselines
Naive reference. The first reference point is the raw sensor depth, together with the same U-Net trained only with plain whole-object masking. This bounds how much any simulation helps at all.
Direct predecessor. The strongest self-supervised competitor is TDCNet with its whole-object erosion masking (Self-Supervised Learning for Transparent Object Depth Completion Using Depth from Non-Transparent Objects). It is the closest method and the only one trained without transparent ground truth. That paper reports about 70% of supervised performance on TransCG and improvements at 5% labels, but the file gives no absolute metric value for it. So there is no reported result found on Lacuna for TDCNet's TransCG error, and we will retrain it ourselves. It gets the same opaque source frames, the same backbone size, the same number of training steps, and the same hyperparameter search budget as our method.
Supervised reference. For the supervised ceiling on single-view TransCG, we use FDCT. The file reports that it beats DFNet by at least 16% and has a mean absolute error (MAE) of 0.010 m on TransCG, as cited in Transparent Object Depth Completion. That multi-view method reports 0.006 m. It uses extra views, so we report it for context rather than as a like-for-like target.
Criteria for beating the baselines. Our fully self-supervised model counts as a clear win if it closes a meaningful fraction of the gap between TDCNet and FDCT's 0.010 m. It also wins if it matches TDCNet on overall MAE while clearly improving seam-boundary accuracy and translucent-cover error.
Ablations. Four variants isolate each ingredient. The first uses part-structured holes only. The second uses compositing only. The third uses both ingredients with plain skips. The fourth uses both with standard gates that lack mask and seam inputs. This last variant separates the benefit of gating in general from the benefit of seam-aware gating.
Why not just…?
Why not just add attention gates to TDCNet and keep its masks? The gates would have nothing to learn. With whole-object holes and the object's own texture visible inside them, there is neither background bleed to suppress nor interior seams to protect. The "both ingredients, plain skips" and "gates, original masks" ablations test this directly. We expect gates alone to be roughly neutral.
Why not just use random multi-blob or patch masks, which also create interior structure? This is the real threat. The predecessor paper reports that object-aware masking significantly outperformed Masked Autoencoder pre-training, which reconstructs randomly masked patches. That result only compares whole objects against random patches, however. A random subdivision of the object mask could match our geometry-verified parts. We include exactly this control: the same hole budget, with interior splits placed randomly instead of at verified depth seams. If it matches us on the seam metric, the part-whole component is not doing work, and we would report the result as a compositing-plus-gating contribution.
Why not just use Depth Anything, zero-shot or fine-tuned on the 5% labels? Depth Anything is trained on web RGB images, and the file notes that it tends to predict the background seen through glass. It is still a strong prior. We will run it with scale alignment to valid sensor pixels, both zero-shot and fine-tuned on the same 5% TransCG split. If it wins at 5% labels, our contribution narrows to the label-free regime, and we would say so.
Why not just render synthetic glass as ClearGrasp does? ClearGrasp trains on 50,000 ray-traced images. Synthetic-to-real transfer still collapses under clutter and translucent covers, where ClearPose reports ADD-S (a 6D-pose accuracy measure) as low as 13% (ClearPose). We will fine-tune the same network on ClearGrasp synthetic data as an extra baseline and compare its performance on ClearPose's hard scenarios.
Proposed Experiments and Decisive Checks
We train on opaque objects from TransCG training scenes without using any transparent depth labels. We then evaluate on TransCG transparent regions using MAE, root mean squared error, and relative error, with the ClearGrasp Real-Novel split as a transfer test. We also run the 5%-label fine-tuning protocol from the predecessor paper.
No existing benchmark has part labels for glass. We therefore define a seam metric from existing ground truth. It is the boundary F-score of predicted interior depth discontinuities against ground-truth depth jumps larger than a fixed threshold inside transparent object masks. We also report MAE within a narrow band around those jumps.
The decisive test is ClearPose's translucent-cover scenario, where an object sits inside another transparent layer. The claim predicts larger relative gains there than on single-layer scenes. We report depth MAE and, following ClearPose, downstream pose ADD-S. For context, Transparent Object Depth Completion reports a 28.2% MAE reduction over TranspareNet in this scenario.
Two outcomes would falsify the proposal. The first is the random-split control matching our seam F-score. The second is gains that appear only in aggregate MAE and not in the seam band or the translucent-cover scenario. Either would mean the benefit comes from generic masking diversity rather than from part structure.
References
- Self-Supervised Learning for Transparent Object Depth Completion Using Depth from Non-Transparent Objects
- Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM
- Guided SAM: Label-Efficient Part Segmentation
- Adaptive Slot Attention: Object Discovery with Dynamic Slot Number
- FDCT: Fast Depth Completion for Transparent Objects
- Attention gated networks: Learning to leverage salient regions in medical images
- Self-Supervised Attention Learning for Depth and Ego-motion Estimation
- Transparent Object Depth Completion
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
- Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation
- ClearPose: Large-scale Transparent Object Dataset and Benchmark