Registering Incomplete 3D Sub-Maps by What Each Camera Saw Through

When a 3D scene is rebuilt from a few unposed photos or depth scans, the system first builds small local 3D pieces and then has to work out how those pieces fit together. Each piece covers only what one sensor could see, so two pieces of the same room often overlap by a third or less. Current alignment methods score a candidate fit by how well the full contents of one piece match the full contents of the other. With only partial overlap, the best-scoring fit can be the wrong one: a mirrored or shifted placement that pairs up geometry that was never shared. This proposal argues that many wide-baseline failures are failures of the scoring rule itself, not of the search that tries to find its best value. More careful search, including search with a proof of global optimality, will not fix them.

The fix uses a piece of evidence that every depth sensor provides for free and that current alignment methods throw away: the empty space a camera looked through. If camera A saw a clear line of sight through some region, then any placement of piece B that puts B's surfaces in that region is impossible. Regions that neither camera saw carry no evidence either way. We propose an alignment score built on three labels, observed surface, observed empty space, and unknown. It rewards surfaces landing on surfaces, robustly penalizes surfaces landing in seen-through space, and ignores unknown space. We pair this score with a global search that can certify its result. The core claim is that, for low-overlap partial reconstructions, the global optimum of a full-matching score is frequently the wrong pose, and that a visibility-aware score turns those cases into ones where the global optimum is the right pose.

Background and the gap

Pose-free reconstruction mostly refines camera poses by local gradient descent on rendering error. NoPe-NeRF trains a neural radiance field (a network that maps 3D points to color and density) jointly with poses, using monocular depth priors. Look Gauss, No Pose does the same inside 3D Gaussian Splatting, a scene representation made of many small colored 3D blobs. The research file reports that this local refinement converges reliably only within about 15 degrees of initial pose error. Both methods motivate this work: they need the substantial frame-to-frame overlap that sparse, wide-baseline capture removes.

RegGS is the closest existing work. It predicts local Gaussian sub-maps with a feed-forward network, treats each sub-map as a Gaussian mixture, and aligns them by minimizing a Mixture 2-Wasserstein distance. That distance is an optimal-transport cost that moves all of one mixture's mass onto the other. RegGS computes it with entropy-regularized Sinkhorn iterations and then refines photometrically. Full-mass transport is exactly the assumption that breaks under partial overlap, because mass from unshared regions must still be moved somewhere. RegGS also searches locally and offers no certificate.

The certifiable solvers supply the search machinery we reuse, but they work on sparse inputs. TEASER certifies rigid registration from 3D keypoint correspondences using truncated least squares. Shonan Rotation Averaging certifies rotation graphs. GOSMA runs branch-and-bound, a search that splits pose space into cells and discards cells whose best possible score is provably worse than one already found. GOSMA aligns mixtures without correspondences, but it assumes a complete 3D model. Our setting removes that assumption.

The free-space idea comes from EnerGS. EnerGS partitions space into occupied, free, and unknown regions from LiDAR. It uses a robust Welsch penalty on occupied space and a barrier on free space, but only to place Gaussians under known poses. We move that partition from reconstruction into registration, where the poses are the unknowns.

Two further lines of work constrain the design. Learning 3D Shape Completion From Laser Scan Data With Weak Supervision shows that raycast free-space likelihoods narrow the synthetic-to-real gap on real KITTI scans: completeness error drops to 9.1 cm, against 17.6 cm for a fully supervised synthetic baseline. That supports free space as a signal that survives the move to real sensors. SplAttN warns that learned completion tends to collapse toward category templates. That warning matters for the "complete first, then register" alternative discussed below.

Core Claim

Each sub-map is built in its own camera frame, so its free space can be carved along its own sensor rays before any global pose is known. Write $\mathcal{G}_B$ for the Gaussians of sub-map $B$, with centers $\mu_i$. Write $o_A(x)$ for the soft occupancy of sub-map $A$ at point $x$, and $f_A(x)$ for its soft "seen-through" field, which is near 1 where some ray of $A$ passed through $x$ before hitting a surface. For a rigid or similarity transform $T$, the score is

$$E(T) = \sum_{i \in \mathcal{G}_B} w_i \Big[ -\,\rho\big(o_A(T\mu_i)\big) + \lambda\, \psi\big(f_A(T\mu_i)\big) \Big] + (\text{same with } A \text{ and } B \text{ swapped}).$$

Here $w_i$ is the opacity of Gaussian $i$, and $\rho$ is a bounded reward for landing on observed surface. $\psi$ is a truncated penalty, in the spirit of truncated least squares, so that a few real-sensor floaters cannot veto a correct pose. Points in unknown space give $o_A \approx f_A \approx 0$ and contribute nothing. This is the property that keeps unshared geometry from biasing the result. Because both terms are bounded and Lipschitz in $T$, meaning they change by at most a fixed amount when $T$ changes slightly, branch-and-bound gives valid bounds over pose cells. The search can therefore return a certified global optimum to within a tolerance.

The claim has two parts. First, we can measure "objective error", the rate at which the global optimum of a score is the wrong pose. For full-mass transport this rate grows sharply as overlap falls; for $E$ it stays low. Second, once the score is right, certified global search removes the remaining "search error" that local methods suffer beyond roughly 15 degrees.

Research Program

The first stage is diagnostic and stands on its own. For pairs of sub-maps with known ground-truth relative pose, we evaluate each candidate score densely over rotation and translation and record two quantities. Search error asks whether local optimization from a given initialization fails to reach the score's global minimum. Objective error asks whether that global minimum lies more than 5 degrees or 5% of scene scale from the true pose. We run this for the RegGS transport distance, a symmetric robust overlap score with no free-space term, and $E$. The overlap axis is controlled by subsampling frames. This stage tells us what the field needs: if transport-based scores have low objective error, better search is enough and the visibility term is unnecessary.

The second stage builds the method. Free-space fields come from carving the depth maps that sub-maps already provide: sensor depth for RGB-D and LiDAR, predicted depth for RegGS-style feed-forward sub-maps. Fields are stored as coarse sparse voxels so that bounds are cheap. Search runs branch-and-bound over rotations with translation nested, following GOSMA's structure. For monocular sub-maps, scale is handled by an outer one-dimensional search. Graduated non-convexity, the strategy of solving an easy smoothed version of a problem first and then sharpening it as TEASER++ and Shape# do, sets the truncation level of $\psi$. Multi-view graphs are assembled by certifying pairwise rotations and averaging them with Shonan. RegGS's photometric refinement then finishes the alignment.

The third stage targets sensor realism. We inject structured corruptions following Sample-adaptive Augmentation: distance-dependent dropout, beam divergence, and specular floaters placed in free space, the last being the worst case for $\psi$. We then compare the ranking of methods on synthetic data against their ranking on real ScanNet and KITTI data, to test whether conclusions transfer.

Baselines

The naive reference is sequential local photometric pose optimization as in Look Gauss, No Pose, initialized at identity. It reports 31.24 PSNR on Tanks and Temples, a dense-overlap setting. Beating it at low overlap is expected and is not by itself evidence for the proposal.

The strongest existing method is RegGS, which reports an absolute trajectory error (ATE) of 0.023 on 8-view RealEstate10K and 28.66 PSNR on 16 views. We will run its released pipeline with identical feed-forward sub-maps, identical photometric refinement, and the same per-pair compute budget. Beating it means lower ATE and a higher fraction of pairs within 5 degrees at overlaps below 30%, while staying within noise of 0.023 ATE at its native 8-view setting.

The key ablation is RegGS's transport distance under exhaustive global search with the same compute as our branch-and-bound. It keeps the score fixed and fixes only the search. A second ablation is $E$ with $\lambda = 0$, which removes free space and isolates its contribution. A third is $E$ with local optimization only, which isolates the value of certification.

For real RGB-D, we compare against TEASER++ on correspondences from local geometric descriptors computed on Gaussian centers, given the same inputs. TEASER reports up to 98.6% registration success on 3DMatch, so it is a serious competitor on depth data. On ScanNet, NoPe-NeRF's ATE of 0.030 provides a dense-video reference point.

Why not just…?

Why not just run global search on the existing transport distance? This is the most direct threat, and the diagnostic stage exists to settle it. If dense search over RegGS's score yields low objective error at 10–30% overlap, our free-space term adds nothing and the honest conclusion is "certify RegGS." The prediction we expect to fail if we are wrong is the reverse: full-mass transport's global optimum drifts toward placements that maximize total overlap, such as mirrored rooms or symmetric object halves, precisely when overlap is small.

Why not just complete each sub-map first, then register with a full-matching score? If completion were accurate, full-mass matching would be valid. But SplAttN documents that completion networks drift toward category templates, and invented symmetric back-halves create exactly the false symmetric fits we worry about. We will test this directly by running a PCN-style completion network on each sub-map before RegGS alignment. If that pipeline matches ours, completion is the simpler answer and we will report it.

Why not just use keypoints with TEASER++? On textured, metric RGB-D data this may well work, and 3DMatch results suggest it can. It weakens where descriptors are unreliable: predicted-depth sub-maps from images, low texture, and sparse LiDAR. We expect a crossover along those axes rather than uniform dominance, and we will report where that crossover falls.

Why not just run SfM? The file documents SfM failing on sparse views, low texture, and degraded sensing, as in Dark3R. We include it as a reference and report its failure rate per overlap bin.

Proposed Experiments

Pose accuracy is measured on RealEstate10K and ACID at 2, 4, 8, and 16 views, on ScanNet RGB-D at frame strides chosen to give overlap bins from 10% to 60%, and on KITTI LiDAR sequence pairs. Metrics are ATE, the fraction of pairs within 5 degrees and 5% translation, and novel-view PSNR after refinement. ACID has no reported number on Lacuna that we found, so we will establish baselines ourselves. Controlled synthetic tests on rendered ShapeNet objects with clean versus structured noise measure objective error under known symmetry.

Three outcomes would be decisive. The first is objective error plotted against overlap for each score. The second is whether $E$ under certified search beats transport under equal-compute global search below 30% overlap. The third is whether $E$'s advantage survives specular floaters on real ScanNet data. If the free-space penalty collapses under real outliers despite truncation, the claim fails on real sensors, and the result reduces to a synthetic curiosity.