A depth sensor such as LiDAR records only the side of an object that faces it, so the result is a partial point cloud. Point cloud completion asks a network to fill in the hidden rest. A popular variant, view-guided completion, also gives the network an RGB photo of the object, sometimes taken from a different angle. A photo from another angle can show parts the scan missed. Recent systems have started running a pretrained monocular depth estimator (a network that guesses per-pixel depth from one photo) on that image and feeding its output to the completion network. Two problems follow. The guessed depth is only correct up to an unknown scale and offset, and it fails on shiny, transparent, or unusual surfaces. Current pipelines pass it through anyway, unchecked, as a block of latent features.
This proposal starts from a simple observation: in this setting the partial scan is metric ground truth for every pixel where the photo and the scan see the same surface. We use that overlap for three things. It fixes the estimator's scale and offset. It measures where the estimator is wrong. It lets us convert only the trustworthy, scan-unseen part of the photo into explicit 3D "evidence points" for the missing region. The completion network then receives two inputs: the real scan, and a set of image-derived points tagged with a confidence that was computed against real geometry. It no longer receives an opaque image embedding.
Core Claim
Image guidance helps completion only where the image sees surface that the scan does not. Its usefulness there depends on whether the monocular depth for that surface can be trusted. Both conditions can be checked directly from the scan–image overlap, without any learned attention. We claim that scan-calibrated, confidence-weighted lifting of image pixels into 3D has three consequences. It should beat latent fusion of depth features in the region visible only to the image. It should degrade gracefully, back to a pure-3D model, when the image adds nothing or the depth estimator fails. It should also explain the "Image-Free Paradox" reported by A Strong View-Free Baseline Approach for Single-View Image Guided Point Cloud Completion. That paper's image-free model beats the multimodal EGIInet on ShapeNet-ViPC. Our hypothesis is that average Chamfer Distance on that benchmark is dominated by views where the image is redundant with the scan.
Relation to Prior Work
The closest work is Leveraging Pretrained Representations for Cross-Modal Point Cloud Completion. It runs Depth-Anything on the guide image, encodes the depth map with a CNN, concatenates the result with DINOv2 features, and feeds both to EGIInet. That paper motivates ours: it shows foundation depth priors help, but it applies no check on depth errors, and the depth only ever reaches the decoder as features. We differ in three ways. Depth enters as metric 3D points rather than features. Scale is fixed against the scan rather than learned implicitly. Each lifted point carries a reliability estimate.
ProjFormer supplies one part of our method and constrains another. It projects candidate 3D points into calibrated views and gates image features by nearness to the camera. However, projecting a hallucinated rear point into a front-facing image binds front-surface appearance to it. We go the other direction: we lift pixels to the surface the image actually sees. This avoids that occlusion mismatch by construction.
The scale-and-offset fit is borrowed from MonoSDF, which aligns monocular depth per batch inside a surface optimizer. Our version is different in that the fit is anchored to real scan points. The idea that reliability should start at verified measurements and decay away from them comes from Masked Spatial Propagation Network, which works on 2.5D depth maps (depth stored as an image). We carry that principle to the 3D completion setting. Any to Full shows that injecting metric scale into Depth Anything works well for depth maps. It does not complete hidden 3D surfaces, which is what we target.
The latent-fusion line serves as baselines rather than method sources. XMFnet uses cross-attention. EGIInet uses Gram-matrix feature transfer. CDPNet reconstructs a coarse cloud from the image and then fuses it with the scan through patch-based style codes. None of the three fixes metric scale. XMFnet explicitly reports degradation when the image view duplicates the scanned side. That observation is the regime our confidence mechanism is designed to detect.
Research Program
The pipeline has three stages: an alignment step, a confidence step, and a completion network that accepts two kinds of points.
The alignment step uses the known camera pose of the guide image, as ShapeNet-ViPC provides. We render the partial scan into that image with a z-buffer. A pixel counts as an overlap pixel when its ray hits a scan point that is front-most along that ray. On the overlap set $\Omega$, we fit the monocular depth $d_p$ at pixel $p$ to the scan depth $z_p$ with a robust loss:
$$ (s^*, t^*) = \arg\min_{s,t} \sum_{p \in \Omega} \rho\big(s\, d_p + t - z_p\big), $$
Here $\rho$ is a Huber penalty and $s, t$ are the unknown scale and offset. We fit globally and also per image segment, to test whether a single affine map is enough. The leftover residuals $r_p = s^* d_p + t^* - z_p$ on $\Omega$ show where the estimator disagrees with reality.
The confidence step assigns each non-overlap pixel $q$ a weight $c_q$ from two sources. The first is the residuals of nearby overlap pixels, propagated through image space with decay along image edges, in the reliability-first style of MSPN. The second is disagreement between two depth estimators: Depth Anything and the Omnidata model used by MonoSDF. When the overlap is small, the global fit is poorly constrained. All confidences are then shrunk toward zero, and the model naturally falls back to scan-only completion.
Each non-overlap pixel is then lifted to 3D using the corrected depth $s^* d_q + t^*$. The resulting evidence points carry their confidence, a tag marking them as image-derived, and a DINOv2 feature from their own pixel. Because each feature is attached to the surface that produced it, rear-surface points never inherit front-surface appearance.
The completion network is a standard point-transformer completion backbone. We use the XMFnet-style decoder so that comparisons are controlled. The encoder receives scan points and evidence points as one set, with the tag and confidence as extra input channels. Training uses Chamfer Distance to the full ground-truth shape plus one asymmetric term. This term penalizes the distance from evidence points to the output, weighted by $c_q$. Trusted evidence therefore pulls the output toward itself, while untrusted evidence is ignored. Scan points are always kept, following the protective masking in ViPC.
A second deliverable is analytical. For every test pair we compute the complementarity: the fraction of ground-truth surface that is visible from the image view but not from the scan view. We report all methods as a function of complementarity, not only as a benchmark average.
Baselines
The naive reference point is the image-free model of A Strong View-Free Baseline. It reports a mean Chamfer Distance of 1.102 and F-Score of 0.851 on ShapeNet-ViPC, and it reports EGIInet at 1.211 / 0.836. We also train our own backbone with the evidence points removed. This "do nothing with the image" control shares every other component with our model.
The strongest existing multimodal number on ShapeNet-ViPC in the file is CDPNet's: mean Chamfer Distance 1.066 and F-Score 0.869 (CDPNet). The most direct competitor on our specific question is Kale et al. That paper reports a 7% Chamfer improvement over existing methods, but no absolute value was found on Lacuna for Kale et al.'s model. We therefore reimplement its dual-branch encoder (Depth-Anything CNN branch plus DINOv2) on the same backbone.
We also include a ProjFormer-style projective-attention variant with known cameras. It is the strongest geometry-aware fusion design, and it isolates our "lift pixels" choice from ProjFormer's "project points" choice.
Reported numbers do not agree across papers. XMFnet reports its own Chamfer Distance as 1.443, while CDPNet lists XMFnet at 1.367. All baselines will therefore be retrained or re-evaluated under one protocol. That protocol fixes the data splits, point counts, Chamfer scaling, F-Score threshold, and a matched GPU-hour and hyperparameter-search budget.
Two ablations isolate the key ingredient. The first, uncalibrated lifting, lifts all pixels using the network's own learned scale without the scan fit. The second, calibrated lifting without confidence, sets $c_q = 1$ everywhere.
Beating the field means meeting two conditions. First, the overall mean Chamfer Distance must be no worse than CDPNet's under our protocol. Second, Chamfer Distance must be clearly lower than both the image-free model and Kale et al. in the image-only-visible region, with the gap growing as complementarity increases.
Why not just…?
Why not just drop the image, since a pure-3D model already wins? This is the most serious threat. Lin et al.'s result shows that, on the ShapeNet-ViPC average, images have not yet been worth their noise. Their own analysis concedes that pure-3D models cannot recover asymmetric parts visible only in the image. The complementarity-stratified evaluation settles the question. If our model matches the image-free model at low complementarity and beats it at high complementarity, the paradox is explained as an artifact of averaging. If it does not beat the image-free model even at high complementarity, the honest conclusion is that single-view images add little to completion. We would report that conclusion.
Why not just concatenate Depth-Anything features, as Kale et al. do, and let the network learn trust? The network sees no metric reference in feature space, and ShapeNet renders rarely trigger depth-estimator failures. A learned implicit trust will therefore not be trained to handle those failures. We test this on a stress split, described below, in which ShapeNet-ViPC objects are re-rendered with glossy and transparent materials. Kale et al.'s gains should shrink on that split; ours should not.
Why not just fit scale and offset and lift everything, skipping the confidence machinery? This could plausibly work as well on clean renders, so we treat it as a central ablation rather than a straw man. If calibrated-without-confidence matches the full model on both clean and stressed data, we drop the confidence component. The contribution then becomes the simpler claim that scan-calibrated metric lifting beats latent fusion. That claim is still untested in the literature.
Why not just use ProjFormer's projective attention? It already uses geometry. However, it attaches image features to hypothesized rear points that project onto front-surface pixels, and it uses no depth prior to propose unseen surface. Our restricted-region metric directly measures whether lifting pixels outperforms projecting points.
Proposed Experiments
The main benchmark is ShapeNet-ViPC: 13 categories, 24 views per object, with known cameras. We report mean Chamfer Distance and F-Score. We also report Chamfer Distance restricted to ground-truth points visible only from the image view, and a phantom-point rate: the fraction of output points farther than the F-Score threshold from any ground-truth surface.
The first decisive check is the complementarity curve described above.
The second is the material stress split. We re-render ShapeNet-ViPC guide images with glossy and transparent shaders, which are the surface types known to break monocular depth estimators (MP-SfM and Depth-Regularized 3DGS both list such failures). On this split, the confidence weights should fall on the corrupted regions, and the full model's restricted-region error should rise much less than that of uncalibrated lifting or Kale et al.
The third is pose robustness. We perturb the image–scan extrinsics, the camera position and rotation relative to the scan, to measure how quickly the overlap-based fit and the confidences degrade relative to ProjFormer-style projection.
For unseen categories, we follow the EGIInet and Kale et al. protocol.
For real data, we evaluate car completion on KITTI with Minimum Matching Distance (MMD) and Fidelity Distance (FD). The reference number is ProjFormer's reported MMD of 0.48. No reported Fidelity Distance number was found on Lacuna for ProjFormer, and its authors note that FD rose. Our prediction for KITTI is specific: because real LiDAR points anchor the fit, our model should match ProjFormer's MMD while lowering FD. That would mean it adheres more closely to the actual scan instead of drifting toward a canonical car shape.
References
- A Strong View-Free Baseline Approach for Single-View Image Guided Point Cloud Completion
- Leveraging Pretrained Representations for Cross-Modal Point Cloud Completion
- ProjFormer: Point Cloud Completion via Geometric-Projective Transformer and Cross-Modal Semantic Constraints
- MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
- Masked Spatial Propagation Network for Sparsity-Adaptive Depth Refinement
- Any to Full: Prompting Depth Anything for Depth Completion in One Stage
- Cross-modal Learning for Image-Guided Point Cloud Shape Completion
- Explicitly Guided Information Interaction Network for Cross-Modal Point Cloud Completion
- CDPNet: Cross-Modal Dual Phases Network for Point Cloud Completion
- View-Guided Point Cloud Completion