Seeing Through Uncertain Rays: Probabilistic Visibility for Image-Guided Point Cloud Completion

When a 3D scanner captures only the front of an object, a photograph of the same object can show what the scan missed. Image-guided completion networks use this, but to borrow a pixel's appearance for a 3D point they must answer one question: does that pixel actually see this point, or does it see something in front of or behind it? Current methods answer badly. Some project every point onto the image and assume the pixel describes it. Some let attention guess the correspondence with no camera geometry at all. Some feed a single predicted depth map into a convolutional network as if it were a color channel. At object silhouettes and thin parts, a pixel mixes foreground and background light, so a single depth value sits in empty space between the two. These are exactly the places where completion fails.

We propose to make the image's depth explicitly probabilistic and use it to compute visibility. Each pixel gets a two-component distribution over the depth of the first surface along its ray. From this distribution we compute, for any candidate 3D point, the probability that the pixel truly sees it. That probability decides how much image evidence the point receives. It also defines a loss that penalizes completed points the image proves are floating in free space. In two sentences: replace the deterministic "project and copy" step in image-guided completion with a soft, ray-based visibility test built from a per-pixel depth mixture. Then use the same distribution to carve away hallucinated geometry the camera can see through.

Core Claim

The image in view-guided completion carries two kinds of information that current methods only partly use. The first is appearance at the pixels that see a point. The second is free space: every ray passes through empty space before it hits a surface. Both depend on knowing where along the ray the first surface lies, and at boundaries that answer is genuinely bimodal.

Our claim is that a bimodal depth distribution per pixel, of the kind SMD-Nets introduced for stereo, gives a principled visibility probability. This is not a heuristic weight. It improves completion where existing methods are weakest: thin structures, silhouettes, and inputs whose camera calibration is imperfect. We also expect the mixture to beat a single-mode Laplacian or a scalar confidence map specifically at those locations. If it does not, the simpler variant should be adopted.

Relation to Prior Work

Six papers combine images with incomplete scans, and they span the design space this proposal sits in. ViPC first reconstructs a coarse cloud from the image and then refines it. CDPNet also starts from an image-reconstructed coarse cloud, then injects a cross-attended style code into patch decoders. Both inherit the errors of an unconstrained single-image reconstruction.

XMFnet fuses image and point features with cross-attention and no projection. EGIInet forces both modalities through shared transformer blocks aligned by Gram-matrix losses. Neither is tied to camera rays, so neither can know when a pixel is occluded.

ProjFormer goes the other way. It projects points deterministically through known cameras and weights features with a "near-field visibility prior" based on distance to the camera. It then gates between observed features and shape priors. Our visibility probability replaces that heuristic with a quantity derived from the image itself, and it degrades gracefully when the projection is slightly wrong.

Kale et al. is the closest work and the main motivation. It shows that adding a pretrained Depth-Anything depth map to EGIInet helps, reporting a 7% Chamfer improvement. However, it passes depth through a CNN as an image channel, so depth never becomes geometry and its uncertainty is discarded. We reuse the same pretrained depth model but turn its output into a ray distribution that interacts with 3D points.

Three works supply the method rather than the problem. SMD-Nets supplies the bimodal Laplacian head, which removes flying pixels at disparity boundaries. PRISM-SLAM supplies the stance that foundation-model depth should be a weighted prior, not metric truth, and that uncertainty gating is what keeps 3D estimates stable. Latent Uncertainty-Aware Multi-View SDF Scan Completion shows that feeding uncertainty into a completion transformer helps. Its uncertainty, though, comes only from 3D heuristics: distance to the surface and the angle between normal and viewing ray. It uses no images. We bring photometric depth uncertainty into the same role.

Research Program

The system has three parts.

Depth head. On top of a frozen Depth-Anything backbone, we train a small head that predicts, per pixel, two depths $\mu_1, \mu_2$, two Laplacian scales $b_1, b_2$, and a mixing weight $\pi$. It is trained by negative log-likelihood on depth maps rendered from the ShapeNet meshes with the benchmark cameras.

Scale alignment. Monocular depth is only defined up to scale and shift. We fit these per image by robust regression against partial-scan points that project into the image. The residual spread of that fit is added to the Laplacian scales, so poor alignment automatically widens the distributions.

Visibility gating. For a candidate 3D point $x$ that projects to pixel $u$ at camera depth $d(x)$, let $Z_u$ be that pixel's first-surface depth under the mixture. We define the visibility probability

$$v(x) = P\big(|Z_u - d(x)| \le \tau\big),$$

where $\tau$ is a surface-thickness tolerance. This is the probability that the pixel's surface is the point itself rather than an occluder in front of it or a surface behind it.

The image feature lifted to $x$ is a mixture. It weights the bilinearly sampled image feature by $v(x)$ and a learned global shape-prior feature by $1 - v(x)$. This replaces ProjFormer's routing gate input and Kale et al.'s CNN depth channel. To absorb calibration error, $v$ and the feature are averaged over a small pixel neighborhood whose size grows with the alignment residual.

Free-space loss. Every predicted surface point must lie at or behind the first surface its pixel sees. We therefore add

$$\mathcal{L}_{\text{free}} = -\sum_x \log P\big(Z_u \le d(x) + \tau\big),$$

where the sum runs over predicted points that fall inside the image. This loss is small whenever the mixture places substantial mass in front of the point. At a silhouette, a point floating just outside the object is penalized only to the extent that the background mode says the ray passes through. Under a single mean depth, the mean trap would place a phantom surface there and either forgive or wrongly punish the point. The loss needs no ground-truth completion, which matters for real scans.

Backbone. Everything is inserted into the Kale et al. / EGIInet pipeline. Gains can then be attributed to the visibility mechanism rather than to a new completion backbone.

Baselines

The naive reference point is a purely 3D completion network that sees no image. PCN is one, and the file reports a VRC-Net mean Chamfer of 4.598 and a TopNet mean Chamfer of 4.976 (F-score 0.467) on ShapeNet-ViPC. This bounds how much the image is worth at all.

The strongest absolute reported number on ShapeNet-ViPC is CDPNet: average Chamfer Distance 1.066 and F-Score 0.869, measured against XMFnet's 1.367 in CDPNet's own evaluation. XMFnet's original paper reports 1.443 ×10⁻³ and F-Score 0.796. EGIInet and Kale et al. report only relative gains (16% over XMFnet, then 7% over EGIInet), so no absolute value for them was found on Lacuna. For ProjFormer, no reported ShapeNet-ViPC result was found on Lacuna; it reports $CD_{\ell_1}$ 6.34 on PCN and MMD 0.48 on KITTI cars.

The competitor that matters most is Kale et al. It already uses the same pretrained depth model on the same backbone, so any gain we show must come from how depth is used. We will retrain it, EGIInet, and CDPNet with released or reimplemented code on the same splits. Each gets the same epochs, the same hyperparameter search budget, and the same GPU hours as our model. Beating it means a lower mean Chamfer and higher F-Score at 0.01 across three seeds, with non-overlapping seed ranges. The gap should be larger on the thin-structure and silhouette subsets than overall.

A deterministic-projection baseline in the ProjFormer style isolates projection from probability. It projects with Depth-Anything depth, applies a hard z-buffer visibility test, and uses the near-field weighting.

Three ablations isolate the key ingredient by changing one thing each. The first replaces the bimodal mixture with one Laplacian. The second replaces it with a learned scalar per-pixel confidence. The third removes the free-space loss while keeping gated features.

Why not just…?

Why not just add a learned per-pixel confidence map to Kale et al.'s depth channel? This is the real threat. It is cheap, and on clean synthetic renders most pixels are unambiguous. A scalar confidence can say "do not trust this pixel" but cannot say which of two surfaces the pixel sees. Without that, it cannot produce a visibility probability or a free-space likelihood at silhouettes. The scalar-confidence ablation settles the question. If it matches the mixture on the boundary-stratified metrics, we drop the mixture and keep the gating and free-space loss.

Why not just project deterministically with a z-buffer test, as ProjFormer's design suggests? A hard z-buffer from monocular depth inherits the mean trap that SMD-Nets documents. At boundaries it declares points occluded or visible based on a depth that lies on neither surface. A one-pixel calibration error flips the test entirely. The deterministic-projection baseline under controlled calibration noise will show whether soft visibility degrades more gently. If the two curves coincide, the probabilistic machinery is not earning its keep.

Why not just use unconstrained cross-attention with calibration-noise augmentation, as in XMFnet? Attention has no notion of occlusion. ProjFormer reports that it also costs quadratic compute and over-smooths fine structure. We retrain XMFnet with the same noise augmentation for a fair comparison. If it matches us on the noisy-calibration curve, ray grounding is unnecessary for robustness and our contribution reduces to the free-space loss.

Proposed Experiments

Main comparison. The main benchmark is ShapeNet-ViPC across all 13 categories, plus the unseen-category split Kale et al. evaluate on. We report mean Chamfer Distance and F-Score at 0.01.

We add two diagnostics computed from ground-truth rendered depth. The first is a floating-point rate: the fraction of predicted points that lie in front of the true first surface by more than $\tau$ in the guiding view. The second is Chamfer stratified by region type, separating points within a few pixels of the image silhouette and points on thin parts from everything else. Thin parts are identified by local thickness of the ground-truth mesh.

First decisive check. Gains over Kale et al. should concentrate in the boundary and thin-structure strata. They should also correlate with the predicted mixture's bimodality, measured as the entropy of $\pi$. Uniform small gains would suggest the improvement comes from extra parameters rather than visibility reasoning.

Second decisive check. We perturb camera extrinsics by increasing rotations and translations and plot Chamfer against perturbation size for our model, the deterministic-projection baseline, XMFnet with augmentation, and Kale et al.

Viewpoint check. We separate image-scan pairs by how much the image sees the scan's missing side. ViPC notes that images of the already-observed side add little. In those cases visibility gating should fall back toward the shape prior rather than hurt.

Real data. On KITTI cars, using the camera images synchronized with LiDAR, we report MMD and Fidelity Distance. The question is whether the free-space loss, which needs no complete ground truth, reduces the fidelity penalty ProjFormer reports alongside its 0.48 MMD. We will run ProjFormer under the same protocol, since the file gives no reported FD value for it.