Glass cups, bottles and beakers defeat robot depth sensors. Light passes through them, so the sensor reports holes or the depth of the table behind. The most accurate fix so far combines two kinds of evidence. The first is a single-image network that guesses depth from shape cues. The second is multi-view stereo, which compares camera views at many candidate depths and stores the comparison scores in a 3D grid called a cost volume. On TransCG, this hybrid cuts depth error by 40% relative to the fastest single-view method (Transparent Object Depth Completion). It is also expensive. The cost volume grows with image height × width × number of candidate depths, and its authors name this cost as the obstacle to real-time robot use.
This note proposes a different way to make that hybrid cheap. The aim is not to shrink the cost volume everywhere. It is to build it only at the depths and pixels where comparing views can actually tell you something about a transparent object. The insight comes from how glass looks. A transparent object's interior shows the background behind it, so cross-view matching inside the glass confidently recovers the background's depth, not the glass surface's. That is the "global shift" failure. We propose to keep that background match as a cheap, low-resolution signal of where transparency is. The single-view guess and its uncertainty then decide which narrow band of depths to test at each pixel. Full-resolution matching is reserved for object boundaries, where cross-view correspondences on glass are reliable.
Core Claim
A transparency-aware hybrid can match or beat the dense single-view-plus-multi-view model. To do so, it should allocate cost-volume memory by three rules. The first is a per-pixel depth band centred on the single-view estimate, with width set by that estimate's uncertainty. The second is a learned gate that sends full-resolution matching to object boundaries and opaque regions and turns it down in glass interiors. The third is a coarse, deliberately unrestricted matching pass whose "see-through" answer serves as a transparency and background-layer cue rather than as a depth estimate.
The memory argument is simple. A dense volume costs on the order of $HWDC$. Here $H \times W$ is image resolution, $D$ the number of depth hypotheses (typically around a hundred) and $C$ the feature channels per hypothesis. Our volume costs roughly
$$ M \approx HWK\,G + |\mathcal{B}|\,K'\,G + \tfrac{HW}{64} D\,G . $$
In this expression, $K \ll D$ is the per-pixel band size and $G$ is the number of correlation groups; group-wise correlation replaces feature concatenation, which CIDER reports roughly halves memory. $\mathcal{B}$ is the set of gated full-resolution pixels with $K'$ hypotheses each, and the last term is the 1/8-resolution see-through pass. The saving relative to dense comes mainly from the ratio $D/K$. The scientific question is whether accuracy survives that reduction, given that the hypothesis band comes from a network that is sometimes wrong.
Why this differs from the obvious combination
The obvious move is to transplant a coarse-to-fine pyramid into the hybrid, following Cost Volume Pyramid Based Depth Inference for Multi-View Stereo (CVP-MVSNet) and Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching. Both narrow the depth search around the previous, coarser multi-view estimate. On glass, that coarse estimate is exactly the global-shifted background depth, so the cascade would lock the error in at every finer level. Downsampling also blurs the thin refractive edges and highlights that carry the signal. The same problem appears in StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction, which builds its volume at 1/8–1/16 resolution and lists transparent car windows among its failures.
Our proposal centres the search on the single-view prior instead. It treats the multi-view coarse answer as evidence about the background rather than about the object. Gating is driven by boundaries, not by scale.
Transparent Object Depth Completion supplies the method we build on. It injects a dense single-view probability volume into a dense multi-view cost volume and merges the two using confidence maps. We keep its central idea that the single-view prior anchors the search, but replace its dense volumes with prior-shaped sparse ones.
Research Program
The system has two stages, trained end to end on the same data as the strongest baseline.
The first stage is a fast single-view network in the style of FDCT: Fast Depth Completion for Transparent Objects, which runs at 70 FPS with 0.39M parameters. It predicts four things per pixel: depth, an uncertainty in the form of a spread, a transparency mask, and a split boundary map. The boundary head follows Internal-External Boundary Attention Fusion for Glass Surface Segmentation. That work separates pixels just inside a glass edge, which carry refraction, from pixels just outside, which carry frame and context, and shows the split improves segmentation. We use the split so that external-boundary pixels, which sit on opaque context, can be matched at full resolution. Internal-boundary pixels are treated more cautiously.
The second stage builds three volumes. A coarse 1/8-resolution volume covers the full depth range with no prior; its peak depth and peak sharpness are the see-through cue. A per-pixel band volume of $K$ hypotheses spans the prior mean ± a learned multiple of the predicted spread. A full-resolution volume is evaluated only at pixels the gate selects. The gate is a small network that reads four inputs: the boundary maps, the mask, the gap between the prior depth and the see-through depth, and the sharpness of the see-through peak. A large gap with a sharp see-through peak indicates glass over a textured background. A flat see-through peak indicates a textureless background. That second case is the "uniform background" problem noted by Phillips et al., where refraction has nothing to distort; the gate should then trust the prior alone.
Regularisation uses lightweight 2D aggregation over each pixel's band, not 3D convolution over a full grid. The output is depth plus an optional background-layer depth. The background layer matters for the translucent-cover scenes, where two transparent layers stack.
The analytic part of the program asks one question directly: does cross-view matching inside glass help or hurt object depth once a good prior exists? We measure this by sweeping the gate's interior weight while holding everything else fixed. Phillips et al. and Glass Surface Detection Method Based on Visual Distortion argue that refraction inside the object is informative. If they are right, interior matching should not be suppressed, and the gate should learn to keep it on. That outcome would revise the design, not break it.
Baselines
The naive reference points are the raw sensor depth and a single-view completion network alone. For the latter we use our own first stage and FDCT: Fast Depth Completion for Transparent Objects, retrained on our splits. These define the floor and show how much multi-view evidence adds. A second classical reference fuses per-frame single-view depths by plain concatenation. Tabletop Transparent Scene Reconstruction via Epipolar-Guided Optical Flow with Monocular Depth Completion Prior (D-EOF) showed this produces noisy, inconsistent geometry on ClearPose. We include it because it is cheap and a reviewer will ask for it. We also report D-EOF itself, which tracks sparse boundary landmarks and refines them with bundle adjustment, reaching an average reconstruction accuracy of 1.14 cm on ClearPose. It is the strongest memory-light competitor, although it yields sparse geometry rather than dense depth.
The strongest competitor is the dense hybrid of Transparent Object Depth Completion. It reports a TransCG mean absolute error of 0.006 m, against FDCT's 0.010 m. On ClearPose's Translucent Cover set, it reports a 28.2% error reduction relative to TranspareNet, but the file gives no absolute value. It is the only method that already fuses a single-view prior with a multi-view cost volume on transparent objects, so it is the competitor to beat. No reported memory or latency figure for it was found on Lacuna, so we will reimplement it with the same backbone, views, training schedule and tuning budget. Peak GPU memory and latency for both systems will be measured on one GPU, and its published TransCG error is the check that our reimplementation is faithful.
Two matched-memory variants of this baseline isolate our contribution. In the first, the dense hybrid is run at reduced resolution or with fewer depth hypotheses until it fits our memory budget. In the second, the dense multi-view volume is replaced by a CVP-style cascade that narrows around the coarse multi-view estimate rather than around the prior.
The ablations remove one ingredient at a time. They replace the uncertainty-adaptive band with a fixed-width band, disable the boundary gate in favour of uniform full-resolution or uniform coarse matching, and drop the see-through cue from the gate's inputs.
Beating the strongest baseline means either of two outcomes. The first is equal or lower transparent-pixel error than the dense hybrid on TransCG and on ClearPose's hard subsets at a substantially smaller peak memory; we target at least a 4× reduction, which is plausible from the $D/K$ ratio. The second is clearly lower error than both matched-memory variants at the same memory.
Why not just…?
Why not just use a single-view network such as FDCT and skip cost volumes? It is fast. However, Transparent Object Depth Completion reports that single-view models smear depth where objects touch, and that adding multi-view matching cuts TransCG error from 0.010 m to 0.006 m. Separately, ClearPose: Large-scale Transparent Object Dataset and Benchmark reports pose accuracy falling to 13% ADD-S under translucent covers, and it ties pose quality to depth quality. Our single-view baseline row measures the gap directly on the Heavy Occlusion and Translucent Cover subsets. If the gap there is negligible, the proposal is not worth its complexity.
Why not just bolt a CVP-style pyramid onto the dense hybrid? This is the most natural efficiency move. We expect it to fail on glass, because each level narrows around a coarse multi-view estimate that, inside transparent regions, is the background depth. The matched-memory cascade baseline tests this. If it matches our method, then prior-centring and gating are unnecessary, and the right paper is the simpler one.
Why not just shrink the dense hybrid: lower resolution, fewer hypotheses, group-wise correlation? This is the real threat. If transparent depth error is dominated by the prior rather than by matching detail, a uniformly downsized dense model may lose little. The matched-memory downsized baseline decides it. We will also report error restricted to boundary pixels, where uniform downsampling should hurt most according to StereoNet's own failure analysis. If the downsized model ties us everywhere including boundaries, we will report that the dense hybrid simply over-provisions memory, which is itself a useful result.
Why not just fuse per-frame single-view depths into a TSDF? D-EOF tested direct concatenation on ClearPose and found it noisy and inconsistent, because nothing enforces agreement between views. We include it as a reference row, but we do not expect it to compete.
Proposed Experiments
Training and evaluation use TransCG and ClearPose. On ClearPose we report each adversarial subset separately: New Backgrounds, Heavy Occlusion, Translucent Covers, Opaque Distractors, Liquids, and Non-planar Surfaces. Pretraining uses Syn-TODD, whose multi-view annotations suit the multi-view stage. Metrics are mean absolute error, root mean squared error and threshold accuracies on transparent pixels, error on internal and external boundary bands, and peak training and inference GPU memory and latency at fixed resolution and view count. Downstream, we run the ClearPose pose pipeline on our completed depth and report ADD-S, with emphasis on Translucent Covers.
Four checks decide the claim. The first is the matched-memory comparison against both the downsized dense hybrid and the multi-view-centred cascade. The second is the interior-gate sweep, which shows whether matching inside glass helps or hurts once a prior exists. The third is a controlled background-texture test. We split ClearPose test frames by the local texture energy behind each object and check that the gate shuts off see-through matching on textureless backgrounds without accuracy collapsing. The fourth is a failure analysis of uncertainty calibration, measuring how often the true depth falls outside the predicted band. If calibration is poor, the narrow band is where the method will break, and widening it trades directly against the memory saving.
References
- Transparent Object Depth Completion
- Cost Volume Pyramid Based Depth Inference for Multi-View Stereo
- Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
- StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction
- FDCT: Fast Depth Completion for Transparent Objects
- Internal-external boundary attention fusion for glass surface segmentation
- A novel stereoscopic cue for figure-ground segregation of semi-transparent objects
- Glass Surface Detection Method Based on Visual Distortion
- Tabletop Transparent Scene Reconstruction via Epipolar-Guided Optical Flow with Monocular Depth Completion Prior
- ClearPose: Large-scale Transparent Object Dataset and Benchmark