Networks that fill in the missing parts of 3D scans are almost always trained on synthetic CAD models, because a real sensor cannot see the back of an object and so cannot provide a complete ground-truth shape. On real LiDAR or depth scans, these networks tend to produce a plausible but generic shape. The overall silhouette is roughly right. The filled-in surface, however, is smooth and template-like, and does not match the object's real roughness, wear, or odd proportions. Yet the observed half of a real scan often already contains the missing geometry. A chair has four legs, a car has two wheels per side, and an industrial truss repeats the same joint many times. This proposal uses the scan's own repeated surface patches as the source of fine geometry. The synthetic-trained network keeps only the job it transfers well, which is saying roughly where the missing surface lies.
The mechanism comes from classical image denoising. Methods in that line gather many similar patches from distant places in an image ("non-local patch grouping") and stack them into a matrix. Because the patches are near-copies, the matrix is nearly low-rank, and noise or missing entries can be removed by keeping only its dominant structure. We apply this to point clouds by flattening each small surface patch into a height map over its local tangent plane. Patches in occluded regions become partially empty height maps. Grouping them with fully observed look-alike patches turns shape completion into low-rank matrix completion. The flattening step is a projection, and it loses information wherever the surface folds back over itself. We treat that information loss as an explicit validity constraint that governs patch size, rather than ignoring it.
Core Claim
A post-hoc, training-free step can do two things for a domain-adapted completion network. It keeps the network's coarse output, and it replaces the network's surface detail in unobserved regions with a weighted low-rank completion of grouped tangent-plane height-map patches drawn from the same real scan. We claim this reduces the sim-to-real gap in fine geometry. The step should give its largest gains exactly where synthetic priors hallucinate most: on real scans with repeated substructures and real sensor noise. It should give little gain on reflection-symmetric shapes that simple mirroring already handles.
This differs from the closest work in specific ways. Syn-to-Real Domain Adaptation for Point Cloud Completion via Part-Based Approach splits a partial input into semantic parts and completes missing parts with learnable part tokens. Those tokens still carry synthetic-domain knowledge, whereas we copy measured real geometry. DAPoinTr and DAPointMamba align synthetic and real features at the token and sequence level. They motivate our starting point, since they are the best available coarse predictors, but they never use self-similarity within a scan. Non-Local Part-Aware Point Cloud Denoising groups self-similar patches in deep feature space with graph attention, but only moves existing points and cannot fill voids. Local Frequency Interpretation and Non-Local Self-Similarity on Graph for Point Cloud Inpainting is closest in spirit: it copies self-similar "source cubes" into holes through a quadratic program. It has no learned shape prior to decide where large missing regions go, uses one source per hole rather than a low-rank group, and does not address the synthetic-to-real setting.
Research Program
The pipeline has three stages. First, a completion network adapted from synthetic to real data predicts a coarse complete shape. We use DAPointMamba as the default and DAPoinTr as a check. From this prediction we place seed points and estimate a tangent frame at each seed. This is the same tangent-plane parameterization that PUGeo-Net uses for upsampling. The difference is that PUGeo-Net needs observed neighbors, while our frames come from the coarse prediction in regions with no observed points.
Second, around each seed we sample a $p \times p$ height map of real observed points. Pixels with no observation are marked missing and filled with the network's predicted heights as a weak prior. Each patch carries a projection-validity score: the fraction of its pixels where the surface is single-valued over the tangent plane. We compute this by checking for multiple depth returns per pixel, which is the small-scale analogue of the vertical collapse documented for bird's-eye-view projections by Domain Adaptation for Vehicle Detection from Bird's Eye View LiDAR Point Cloud Data. Patches below a validity threshold are split into smaller radii. This makes the projection step lossless up to the chosen resolution, rather than silently merging folded surfaces.
Third, each partially observed patch is matched against observed patches anywhere on the object. Matching uses a rotation-normalized distance computed on the shared observed pixels plus the predicted pixels. The $K$ best matches are stacked as columns of a matrix. The completed patch matrix $X$ solves
$$\min_{X}\ \|P_{\Omega}(Y - X)\|_F^2 + \beta\,\|P_{\bar\Omega}(\hat Y - X)\|_F^2 + \sum_i w_i\,\sigma_i(X).$$
In this objective, $Y$ holds the observed heights and $\Omega$ is the set of observed pixels. $\hat Y$ holds the network's predicted heights on the unobserved set $\bar\Omega$, and $\beta < 1$ down-weights them. $\sigma_i(X)$ are the singular values of $X$, and $w_i = c\sqrt{K}/(\sigma_i + \epsilon)$ is the weighting from Weighted Nuclear Norm Minimization. That weighting shrinks weak, noise-dominated components hard while sparing the dominant components that encode edges and thin structures.
The mask breaks WNNM's closed-form solution. We therefore solve the problem by alternating imputation with the generalized singular-value soft-thresholding step, a standard approach. Completed patches are lifted back to 3D through their tangent frames and averaged where they overlap. As a side effect, the same low-rank step suppresses real sensor noise on observed regions.
The analysis side asks when self-similarity helps and when it hurts. We will measure, per object, the effective rank of the grouped patch matrices, and relate it to per-object gain over the network. We will also track how often false matches occur, since grouping unrelated structures is the known failure of Euclidean block matching under heavy noise.
Baselines
The naive reference point is the adapted network's output used as-is. For object-level synthetic-to-real completion this means the part-based method of Yang et al., which reports on USSPA a Chamfer Distance "reduced to 6.99" from a previous best of 9.01 and an F1 "increased to 80.5". On 3D-FUTURE the reference is DAPointMamba, which reports a Chamfer Distance of 20.40 against DAPoinTr's 22.35. On KITTI cars, DAPoinTr reports a unidirectional Chamfer Distance of 0.45. For RealPC, no reported result was found on Lacuna for DAPoinTr, DAPointMamba, or the part-based method; the file reports only that ODGNet, PoinTr, and SnowflakeNet show high errors there. We will run all of these ourselves.
Our method is applied on top of exactly these checkpoints. The comparison therefore isolates the patch step: same inputs, same coarse predictor, no extra training data. Beating the strongest baseline has two parts. First, we must reduce Chamfer Distance and raise F1 on USSPA and 3D-FUTURE beyond the variance across three seeds. Second, we must win clearly on the masked-observation fidelity metric described below, because Chamfer Distance to CAD ground truth partly rewards synthetic-looking output.
Four further baselines test specific pieces of the claim. The first is the classical inpainting method of Fu et al., run on the same scans; it tests whether the learned coarse prior is needed at all. The second replaces our explicit low-rank completion with learned feature-space grouping in the style of Huang et al.. It is extended to predict displacements for points initialized from the network output and trained on the same synthetic data. The third runs PUGeo-Net upsampling on the network output, which tests whether we merely add density. The fourth covers ablations. One ablation replaces nearest-neighbor grouping with random patch groups of the same size, isolating non-local similarity. Another replaces weighted with plain nuclear norm; WNNM reports a 1.3 to 2 dB advantage over plain nuclear norm in images, and we will check whether that advantage carries to geometry. A third removes the projection-validity splitting.
Why not just…?
Why not just fine-tune the completion network on each real scan at test time? One could self-supervise the network by masking parts of the observed scan and training it to reconstruct them. This is a real threat because it also injects real-scan information. It still routes all geometry through a decoder trained on synthetic data, which Revisiting Point Cloud Completion shows fails on real topology and sparsity. It also costs gradient steps per scan. We will run it with the same masking protocol and a matched wall-clock budget. If it matches our fidelity numbers, the explicit matrix step is not justified, and we would recast the contribution as a regularizer inside test-time tuning.
Why not just mirror the observed half across a detected symmetry plane? For cars and most furniture this may recover much of what we recover. We expect it to be competitive on KITTI cars and USSPA. The proposal predicts a clear advantage only on repeated but non-mirrored structure and on scans where the mirror image is also occluded, which is common in RealPC's industrial categories. We will report results split by category. If mirroring ties us everywhere, the self-similarity story reduces to symmetry and the claim fails.
Why not just use learned non-local attention, which already groups similar patches? Attention weights are learned on synthetic data and inherit its biases. The NLU of Huang et al. was designed to move existing points, not to fill voids. The feature-grouping baseline above tests this directly. If it matches us, the value lies in grouping rather than in low-rank completion.
Why not just process a bird's-eye or range projection with 2D networks? Scene-level projections collapse exactly the vertical and occluded structure that completion must recover. GFNet shows that range-view trucks reach only 32.4% mIoU without extra geometric information. Our projections are local and validity-checked, so they avoid this collapse.
Proposed Experiments
The decisive check is a masked-observation protocol on real data. On KITTI cars, ScanObjectNN, and RealPC, we remove a contiguous observed region chosen to mimic occlusion, run every method on the remainder, and measure Chamfer and Hausdorff distance to the removed real points. This gives real-sensor ground truth without CAD models and directly measures hallucination versus fidelity. The prediction that would support the claim is specific: our step lowers held-out error relative to the adapted network, and the gain correlates positively with the measured low-rankness of an object's patch groups. A second prediction is that gains are near zero for objects whose groups have high effective rank. If neither holds, the claimed mechanism is wrong.
Secondary experiments complete the picture. We report standard Chamfer Distance and F1 on USSPA and 3D-FUTURE against the reported numbers above. On RealPC we report CD-L1, CD-L2, and persistent-homology consistency, which tests whether copying real patches preserves loops that synthetic priors break. We also sweep patch radius and validity threshold, measuring accuracy against the fraction of folded patches. Finally, we measure runtime relative to DAPointMamba's reported 3.8 ms inference. Our optimization will be far slower, so the honest framing is offline reconstruction rather than real-time use.
References
- Syn-to-Real Domain Adaptation for Point Cloud Completion via Part-Based Approach
- DAPoinTr: Domain Adaptive Point Transformer for Point Cloud Completion
- DAPointMamba: Domain Adaptive Point Mamba for Point Cloud Completion
- Non-Local Part-Aware Point Cloud Denoising
- Local Frequency Interpretation and Non-Local Self-Similarity on Graph for Point Cloud Inpainting
- PUGeo-Net: A Geometry-Centric Network for 3D Point Cloud Upsampling
- Domain Adaptation for Vehicle Detection from Bird's Eye View LiDAR Point Cloud Data
- Weighted Nuclear Norm Minimization with Application to Image Denoising
- Revisiting Point Cloud Completion: Are We Ready For The Real-World?
- GFNet: Geometric Flow Network for 3D Point Cloud Semantic Segmentation