Depth cameras fail on glass. A transparent cup shows up as a hole, or as a scatter of points from the table behind it. Networks that fill in this missing depth all feed the raw camera-relative depth map into their encoders, so the input changes whenever the camera tilts, even if the scene does not move. Indoor RGB-D work solved this tilt problem years ago with HHA, a three-channel re-encoding of depth: horizontal disparity, height above the ground, and angle between the surface and gravity. HHA was never used on transparent scenes, because computing it needs good surface normals, and normals inside glass regions are garbage.
Our observation is that HHA does not need normals from the glass. Gravity and the supporting plane can be estimated from the opaque pixels around the glass, such as the table, the shelf, and the floor. Once that frame is fixed, the height and gravity-angle channels say where a glass object must touch its support and which way is up, and they say it without any valid depth inside the object. We propose to build HHA from opaque pixels only, carry it through the network on a skip path gated by depth validity, and recompute it once from the completed depth. The main empirical question links two of our directions. We test whether much of the measured gap between small convolutional backbones and Swin-style transformers on transparent depth completion is really a gap in input representation, which would let a real-time CNN recover most of the transformer's accuracy.
Core Claim
The claim is that a gravity-anchored input built from trustworthy pixels gives a better inductive bias for transparent depth completion than raw camera-relative depth. We further claim that this benefit is largest for small, fast backbones. If true, two things follow. The backbone ranking reported in this area partly reflects what the depth stream is given rather than only what the backbone can model. And accuracy under camera tilt should improve sharply, because raw-depth models have to learn tilt invariance from data.
This claim is different from all the closest work. RDFNet and JL-DCF use geometric depth encodings with CNN backbones, but only on opaque indoor scenes, where normals are reliable. TODE-Trans, TDCNet and FDCT attack transparent depth completion through the architecture: a Swin encoder, a CNN-transformer dual branch, and fast cross-layer shortcuts respectively. All three feed raw single-channel depth. ClearGrasp and TransNet show that estimated surface normals are more reliable than direct depth on glass. However, they use normals as constraints in a slow energy solver (1.82 s per frame) or as an extra input channel, and never place them in a gravity frame.
Research Program
Robust geocentric frame. The first component estimates gravity and the dominant support plane from opaque pixels only. The opaque set is defined as pixels with valid sensor depth that lie outside a predicted transparency mask, where the mask comes from a small RGB segmentation head. On the opaque points we run a robust plane fit (RANSAC, an algorithm that fits a plane while ignoring outliers) together with the standard HHA gravity-refinement iterations from Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images. With gravity direction $g$ and support-plane offset $d_0$ fixed, the height channel at pixel $p$ with back-projected 3D point $X(p)$ is
$$h(p) = g^\top X(p) - d_0 .$$
This value is computed only where depth is trusted. Elsewhere it is set to a sentinel value, and a validity channel marks which pixels are trusted. The gravity-angle channel is handled the same way. The disparity channel is kept as is, with holes marked. The whole computation runs on the GPU, and we report its latency separately, because the file names HHA preprocessing as a real-time bottleneck.
Validity-gated geometric skip path. The spatial-detail literature warns that plain skip connections pass sensor noise straight into the decoder. To address this, the geocentric channels go through a light convolutional stem, and their features enter the decoder at each scale through a gate:
$$s_\ell = m_\ell \odot \phi_\ell(\text{HHA}) + (1 - m_\ell)\odot \psi_\ell(\text{RGB}).$$
Here $s_\ell$ is the skip feature at decoder scale $\ell$, $m_\ell$ is a learned gate initialised from the downsampled validity mask, and $\phi_\ell$ and $\psi_\ell$ are the geometric and appearance feature extractors at that scale. The intent is that precise boundaries such as table contact lines come from trusted geometry, while glass rims come from appearance.
One re-encoding pass. After a first prediction, HHA is recomputed from the completed depth, now including the glass regions, and a second pass of the shared decoder refines the result. This is the "HHA post-completion" step that the research file lists as unexplored.
Backbone sweep. All three components are placed around interchangeable encoders: an FDCT-style one-shot-aggregation CNN, ResNet-18, and Swin-T as used in TODE-Trans. This allows the representation effect and the backbone effect to be separated in a single factorial design.
Baselines
The naive reference point is each backbone trained on the four-channel concatenation of RGB and raw depth. The training data, loss, schedule, and tuning budget are the same as for our models. For each backbone, the only change between this baseline and our models is the depth representation.
The strongest single-view competitors are TODE-Trans (public code, Swin backbone) and TDCNet (dual branch). We retrain both on our splits with their published recipes, and give them the same number of hyperparameter trials that our models get. For reference numbers, the file reports FDCT at 0.010 m MAE on TransCG. Transparent Object Depth Completion reports 0.006 m, but that method is multi-view, so it is a ceiling rather than a like-for-like competitor. ClearGrasp reports 0.039 RMSE on its real-known test set, and the RGB-D Local Implicit Function reports 0.020 m MAE on real unseen objects. No absolute TransCG numbers for TODE-Trans or TDCNet were found on Lacuna; the file gives only relative gains, such as TODE-Trans's 25% RMSE improvement on ClearGrasp. For that reason, our retrained versions are the operative comparison.
Beating the field means two things. On TransCG, our FDCT-sized model should go below FDCT's 0.010 m MAE at comparable frame rate. And our Swin model should outperform retrained TODE-Trans and TDCNet on MAE and on boundary-band error.
Two ablations isolate the key ingredients. The first computes classic HHA from all pixels, including corrupted glass pixels. This tests whether computing the frame from opaque pixels only is what matters. The second replaces HHA with a gravity-aligned point map, meaning XYZ coordinates in the estimated gravity frame. This tests whether the gain comes from the gravity frame itself or from HHA's specific channel formulas. We expect the point map to perform close to HHA. If it does, we will report the contribution as the opaque-anchored gravity frame, not HHA as such.
Why not just…?
Why not just use a bigger transformer on raw RGB-D? TODE-Trans shows that a Swin encoder helps substantially. Our sweep includes Swin with raw depth, so the question is answered directly. If the representation gain disappears for Swin but stays large for small CNNs, that is still the useful result, because it shows how to get a deployable fast model. If HHA gives no gain at any backbone size, the core claim fails, and we will report that.
Why not just train with tilt and rotation augmentation on raw depth? This is the most serious threat, because augmentation can teach invariance without explicit geometry. We add a baseline that applies synthetic camera-pitch perturbations to raw depth during training: images are reprojected and holes are left in place. The decisive test is a tilt-binned evaluation, described below. If augmentation matches our method in the high-tilt bins, the invariance argument for the encoding is weakened. What would remain is a possible benefit from the support-plane height prior, which augmentation cannot supply.
Why not just add a validity mask channel, or surface normals, to raw depth? A validity mask alone might explain much of the gain from gating. TransNet already shows that adding estimated normals helps. Both are run as input variants with identical architectures. Our claim requires the gravity frame to add value beyond the mask and the camera-frame normals.
Proposed Experiments
We train and evaluate depth completion on TransCG and on ClearGrasp, including its real-known and real-novel splits. We also test cross-dataset transfer by training on ClearGrasp plus Omniverse and testing on TransCG, following the TDCNet protocol. Metrics are MAE, RMSE, REL, and threshold accuracies, plus a boundary-band MAE computed within 5 pixels of transparent mask edges, which is where detail preservation matters. Frame rate is measured on one GPU with HHA computation included.
The first decisive check bins test frames by camera pitch relative to gravity. Pitch comes from dataset poses where these are available, and otherwise from our opaque-plane estimate, which we validate against ground-truth plane fits. We then plot error against tilt for raw depth, augmented raw depth, and our encoding. Our hypothesis predicts a flatter curve for our method.
The second check is the backbone-by-representation interaction. For each backbone we compute the fraction of the raw-depth gap between that backbone and Swin that our encoding closes.
The third check is downstream. Completed depth goes into a fixed pose estimator on ClearPose, and we report ADD-S separately for each of the benchmark's scenarios. ClearPose itself reports best models at about 58% ADD-S in easy scenarios, falling to about 13% under translucent covers. We expect the support-plane prior to help with opaque distractors and new backgrounds. We do not expect it to fix translucent covers, because a single depth value per pixel cannot represent two transparent layers, and stating this limit is part of the result.
Finally, we report how often gravity estimation fails, meaning an error above 5° against ground-truth planes, in scenes with little opaque support. The method's assumption breaks in exactly those scenes.
References
- RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation
- JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection
- TODE-Trans: Transparent Object Depth Estimation with Transformer
- TDCNet: Transparent Objects Depth Completion with CNN-Transformer Dual-Branch Parallel Network
- FDCT: Fast Depth Completion for Transparent Objects
- Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation
- TransNet: Category-Level Transparent Object Pose Estimation
- Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images
- Transparent Object Depth Completion
- RGB-D Local Implicit Function for Depth Completion of Transparent Objects