Closing Sensor-Broken Silhouettes of Transparent Objects

Depth cameras break on glass in a particular way. An active stereo camera projects an infrared dot pattern and matches it between two views. Where the rim of a glass curves away from the camera, the pattern bends, reflects internally, or is hidden from one of the two views, so the raw depth map loses the object's outline in predictable stretches. Current transparent-object depth completion networks learn to fill the inside of the object, and they blur depth across the outline when two glasses touch. The older contour-closure methods from edge detection can join broken boundary pieces into closed loops, but they treat every gap the same way, as generic noise. We propose to simulate the sensor's own failure mechanics to learn where and why a specific camera loses the boundary of a transparent object. That learned model of the gaps then sets the costs of a globally optimal contour-closing step. The closed contours act as hard partitions for depth completion.

In two sentences: the gaps in a transparent object's outline are not random, because they follow the projector-camera geometry and the surface curvature. A simulator of the exact sensor can therefore teach a contour-closing algorithm which gaps to bridge. If this holds, it attacks two documented failures at once: ClearGrasp's random depth assignment inside unclosed boundaries, and the boundary smearing of fast feed-forward completion networks.

Core Claim

Sensor-noise simulators already reproduce transparent-object depth corruption well enough to train restoration networks. DREDS/SwinDRNet simulates the infrared projector, refraction and stereo matching, and raises grasp success on specular and transparent objects from 38.78% to 96.15%. These simulators are evaluated only at the level of whole areas. Nobody has asked whether they reproduce where the boundary disappears.

Our claim is in two parts. First, physically simulated active stereo predicts the location of real silhouette dropouts on transparent objects well above what curvature or image contrast alone predict. Second, using that prediction as the cost of bridging a gap in a closed-contour search yields better object partitions, and better depth near boundaries, than either end-to-end networks or contour closure with hand-tuned gap costs. The edge-fragmentation literature supplies the closure algorithm, sensor modeling supplies the gap statistics, and transparent perception is the regime where both matter.

The closest existing work combines only two of the three pieces. ClearGrasp predicts occlusion and contact boundaries from RGB and feeds them to a global depth solver. Its authors report that the solver becomes indeterminate when boundaries fail to close, and the method uses clean renders rather than a model of the sensor. DREDS models the sensor but has no explicit boundary or closure mechanism. Ratio Contour finds globally optimal closed boundaries from broken edges. It does this by alternating detected fragments with candidate bridging gaps and minimizing a cost-per-length ratio, solved exactly via minimum-weight perfect matching. Its gap cost is a fixed function of gap length and curvature, and it has only been shown on natural images and MRI. Our proposal changes Ratio Contour's gap term, the one ingredient none of these papers touches.

Research Program

The first component is a boundary-resolved sensor simulator. We extend the active-stereo pipeline of Zhang et al., which combines ray tracing with transmissive materials, infrared speckle projection and GPU semi-global stereo matching at over 200 FPS. We extend it with the DREDS-style refraction and multi-path setup, and render the transparent CAD models from ClearGrasp and ClearPose.

The new output is a per-boundary-pixel label: whether the simulated sensor returned valid, correctly placed depth at that silhouette point, and if not, which mechanism caused the loss. The mechanisms are one-view occlusion, grazing-angle refraction, internal reflection, or matching failure. We record the local geometry with each label: surface normal relative to the projector and both cameras, and rim curvature. Where real captures exist, we can tune unknown projector and matching parameters against them with the differentiable simulator of DDS. DDS was built for opaque objects, but we use it only for the sensor-side parameters.

The second component is a learned gap model. For a candidate bridge between two fragment endpoints, a small network estimates the probability that a true silhouette runs through the gap, written $p_\theta(g)$. The fragments come from RGB edges, raw-depth discontinuities and depth-validity borders combined. The network sees the raw depth validity pattern, RGB features along the straight and curved bridge hypotheses, and the sensor-geometry features above: baseline direction relative to the gap orientation, and the expected angle of incidence. It is trained on simulated pairs of fragmented observations and true silhouettes.

The third component is closure and completion. Inside each candidate transparent region from a segmentation network, we build Ratio Contour's alternating graph and minimize

$$ CR(C) = \frac{\sum_{s \in C_{\text{frag}}} w(s) \;-\; \sum_{g \in C_{\text{gap}}} \log p_\theta(g)}{\sum_{e \in C} \ell(e)}, $$

Here $C$ is a candidate closed contour, $C_{\text{frag}}$ and $C_{\text{gap}}$ are its detected fragments and bridged gaps, $w(s)$ is a fragment's edge-strength cost, and $\ell(e)$ is the length of piece $e$. The only change from Ratio Contour is that the hand-designed gap penalty is replaced by the learned negative log-probability. Because the graph is restricted to the fragments near one object, it should be small enough for the polynomial matching solver to run in tens of milliseconds. Whether it does is measured, not assumed.

The extracted closed contours, with occluding and touching pairs kept as separate loops, are passed to a fast completion network. We use FDCT-style or SwinDRNet-style completion. The contours are passed both as an extra input channel and as a hard constraint that forbids smoothing across a closed boundary. For ClearGrasp's solver, they replace the open boundary map, which directly removes its indeterminate case.

Baselines

The naive reference is raw sensor depth, together with the same completion network trained on clean renders plus Gaussian jitter and random dropout masks. This tests whether any structured modeling matters. Quasi-Balanced Self-Training argues that independent random noise of this kind misrepresents real sensor error. The structured-light study Bai et al. reports real-data detection IoU of 0.584 for standard domain randomization versus 0.648 for physics-based simulation. Its setting is opaque industrial parts, so it does not settle the transparent case.

The strongest end-to-end competitor that models the sensor is SwinDRNet trained on DREDS, because it already has the physical noise and lacks only explicit topology. We will retrain it on our simulated data with the same image count and tuning budget, so that any gain is attributable to closure rather than to data. Its reported results are the 30 FPS speed and the 38.78% to 96.15% grasp improvement on its own robot setup. Those are not on our benchmarks, so we rerun it.

The strongest accuracy baseline is the hybrid single-view and multi-view cost-volume network of Zhou et al.. It reports TransCG MAE of 0.006 m, against 0.010 m for FDCT. It also reports a 28.2% MAE reduction over TranspareNet on ClearPose translucent covers, with no absolute value given. Its authors note that it still smears depth between touching objects and is slow. Beating it means matching its overall TransCG MAE within noise, while clearly reducing error in a band of a few pixels around touching silhouettes, at single-view latency.

On ClearGrasp's unseen split we compare against TranspareNet, which reports RMSE 0.032 versus ClearGrasp's 0.044. On real unseen objects we compare against LIDF, which reports MAE 0.020 m. Every model is trained on the same simulated set and evaluated with each benchmark's official protocol.

Three ablations isolate the key ingredient. The first is Ratio Contour with its original length and curvature gap cost on the same fragment graph. The second is our gap model with the sensor-geometry inputs removed, so it has RGB and depth features only. The third is the gap model trained on simulations of a mismatched sensor geometry, with the baseline flipped or the projector moved. If the second ablation matches the full model, the claim that the gaps are sensor-specific is wrong, and the contribution reduces to "learned contour closure."

Why not just…?

Why not just train SwinDRNet or FDCT on the physical simulation and let attention learn closure? This is the real threat. Dense networks reconstruct interiors well, and Zhou et al. report that smearing at touching boundaries persists even with multi-view cues. FDCT's shortcuts, however, can only carry edges that exist in early features. The matched-data SwinDRNet baseline settles it directly. If retraining on our boundary-resolved simulation closes the boundary-band error gap without explicit closure, we will report that the value lies in the simulator labels, not the combinatorial step.

Why not just use Ratio Contour off the shelf on a good learned edge map? Its gap penalty prefers short, smooth bridges. Sensor-induced gaps on glass rims are often long and lie along the stereo baseline direction, where a generic prior is weakest. The first ablation tests exactly this. We expect the difference to concentrate on the gaps the simulator attributes to one-view occlusion and grazing refraction.

Why not just add a contour-detection network trained on real transparent data and threshold its output? ClearGrasp already does this with a dedicated boundary network. It reports that unclosed predictions make its depth solver assign random values, which shows that per-pixel boundary prediction does not guarantee loops. A thresholded network is included as the input-fragment source in all our variants, so its standalone closure rate is reported as a baseline.

Why not just drop active depth and use multi-view or radiance-field reconstruction? Zhou et al. show that multi-view matching locks onto the stable background behind glass, and radiance-field optimization is far from closed-loop speed. Our setting is single-frame RGB-D at robot rates, so we treat these methods as out of scope rather than as competitors.

Proposed Experiments

The decisive first check precedes all modeling of closure. On real STD and ClearPose frames with ground-truth silhouettes from CAD alignment, we measure how well simulator-predicted dropout probability ranks the real silhouette pixels that lack valid depth (area under the ROC curve). We compare this against curvature-only and image-contrast-only predictors. If the simulator does not clearly outperform these, the sensor-specific premise fails, and we stop and publish the negative finding.

The main evaluation covers ClearGrasp (real-known and real-novel), TransCG, TODD and ClearPose. We report overall MAE and RMSE, error restricted to a boundary band, a smearing score (error on pixels between touching instances), the fraction of objects with a closed contour, contour F-score, and end-to-end latency. ClearPose's translucent-cover split is the stress test, with a reported 13% ADD-S for prior pose methods. There, two nested boundaries must be closed as separate loops, and we report both depth error and downstream ADD-S using the dataset's pose pipeline.

The sensor-swap experiment provides the second decisive check. We train gap models on simulated geometry matched to the capture camera and on deliberately mismatched geometry, then evaluate on the same real frames. The sensor-specific claim predicts a measurable drop for the mismatched model, recovered by re-simulating with the correct geometry. The ClearGrasp solver experiment replaces its boundary map with our closed contours and counts how many previously indeterminate regions become determined, together with the resulting depth error.