Telling the Static World from Moving Cliques in Low-Overlap Registration

Registration is the task of aligning two 3D scans of the same place into one coordinate frame. It gets hard when the scans share only a small region, say 10 to 30 percent, because most candidate point matches are then wrong. The best current methods handle this by searching for groups of matches that agree with one another and by borrowing texture from camera images when the geometry is flat and featureless. All of them assume the world holds still between the two scans. When a car drives or a person walks through the small shared region, that assumption fails in a specific way. The moving object's points still agree perfectly with each other, because the object is itself rigid. The matcher therefore finds a clean, self-consistent group of matches that recovers the object's motion instead of the scanner's. In low overlap, this wrong group can easily be the largest or most distinctive one available.

We propose to stop treating a moving object as noise to be filtered out and to treat it as a competing explanation to be rejected. The method generates many rigid alignment hypotheses from image-plus-geometry matches. It then scores each hypothesis by how well it explains the whole co-visible region, using dense image features and depth free-space checks. Disagreement is allowed only in a few compact, object-sized regions, which are declared "moved." The correct alignment explains the background and pays a small price for each mover. The object-aligned alignment leaves the entire background in disagreement, and that disagreement cannot be declared moved because it is not object-sized. Parked cars and other static objects are never masked away, so they remain useful anchors.

Core Claim

In dynamic low-overlap scenes, the dominant failure is that the hypothesis selector chooses the wrong rigid consensus, not that the descriptors are weak. A verification score that combines cross-modal agreement with a sparsity prior on object-sized motion regions will fix most of these failures. It will do so without discarding the static structure that semantic masking throws away. It should also cost nothing on static benchmarks, because there it reduces to a dense agreement check.

Positioning Against Prior Work

Three works supply the pieces we reuse. 3D Registration with Maximal Cliques (MAC) builds a graph whose edges connect matches that preserve pairwise distances. It extracts many local maximal cliques, meaning groups that cannot be grown further, and picks one by a mean-absolute-error score over matches. We keep its hypothesis generation but replace that selection score. Our motivating observation is that MAC's score counts consistency within the match set, and a moving rigid object is fully consistent within itself. Cross-modal feature fusion for robust point cloud registration with ambiguous geometry (CoFF) projects 3D points into calibrated images and fuses pixel-level and patch-level image features to tell apart identical-looking planar surfaces. We use CoFF-style fused descriptors to feed the hypothesis pool. We do not trust projected features during verification without a motion check, because CoFF writes foreground pixels onto occluded background points whenever something moves. ViGG uses visual cliques to bound the search for geometric matches. Like CoFF, it assumes a rigid, static scene, and its authors concede it fails in low texture. Our verification uses depth free-space as a second signal precisely so that low texture does not disable it.

Two works frame the dynamic side and constrain the design. Multi-Model 3D Registration explicitly explains correspondences as several rigid motions. It uses an expectation-maximization loop with a metric gating threshold that keeps spatially separate objects apart, reporting IoU 0.970 against 0.398 for Sequential RANSAC on PASCAL3D+. However, it depends on dense scene flow and high overlap, and it uses geometry only. We borrow its idea of bounding motion clusters by metric extent, but we apply it in image space during verification rather than as a clustering of sparse, low-overlap correspondences. RobustSplat and Split4D show that transient objects corrupt reconstructions and that separating static from dynamic parts works best when it is learned jointly rather than imposed by an external 2D segmenter, whose errors, the Split4D authors argue, propagate. That is our main argument against pre-masking. Both assume posed, densely overlapping video, which is not our setting.

The closest pairwise work, OG-UPCR, uses Gaussian-splat rendering for unsupervised low-overlap RGB-D registration under lighting change, with no dynamic handling. ImLoveNet fuses uncalibrated images through attention and also assumes a static scene. Neither changes how a hypothesis is chosen when the scene contains a second rigid motion, and that choice is our contribution.

Research Program

The pipeline has three stages. First, fused descriptors (a CoFF-style image-geometry encoder) produce putative matches, and MAC extracts the top $K$ maximal-clique hypotheses, with $K$ around 50. The default is to keep MAC's graph unchanged, so any gain can be attributed to selection. Second, each hypothesis $T$ is used to warp the source scan's depth and dense image features, taken from a frozen DINOv2-style backbone as in Split4D, into the target view. From the warp we compute a per-pixel residual $r_T(u) \in [0,1]$ over the predicted co-visible pixels $V_T$. The residual takes the larger of two terms. The first is feature disagreement, one minus cosine similarity. The second is a free-space violation, where a warped source point lands in space the target depth sees through, or the reverse. Third, the target image is split into class-agnostic segments by clustering depth and features, and hypotheses are chosen by

$$T^\star = \arg\max_{T} \sum_{u \in V_T \setminus M_T} \big(1 - 2\,r_T(u)\big) \;-\; \lambda\, n(M_T),$$

where $M_T$ is the union of segments declared "moved," $n(M_T)$ is the number of such segments, and $\lambda$ is a fixed cost per declared segment. A segment may be declared moved only if its metric extent in the target depth is below an object-scale bound $\tau$, the same role the gating threshold plays in Multi-Model 3D Registration. A segment is declared greedily when excluding it improves the score by more than $\lambda$. Agreeing pixels count as +1, disagreeing pixels as −1, and declared pixels as 0. As a result, a hypothesis aligned to a moving car cannot excuse the disagreeing background, while the correct hypothesis excuses the car at the cost $\lambda$. Static objects that agree, such as parked cars, add positive evidence instead of being removed.

Two analyses accompany the method. The first audits failures. On every failed pair for each baseline, we record whether the chosen clique's inliers lie mostly on a mover, using ground-truth motion labels. This tells us whether wrong selection is really the dominant failure; if it is not, the proposal's premise is wrong. The second measures which residual channel carries the decision in each regime. We expect free-space to dominate outdoors and image features to dominate in indoor planar scenes.

Baselines

The naive reference is MAC with GeoTransformer descriptors and its own error-based selection, run with no dynamic handling. It is the natural "do nothing" point, and MAC reports 78.9% registration recall on 3DLoMatch and 99.46% success on KITTI. The strongest existing static method is CoFF, which reports 81.6% registration recall on 3DLoMatch and more than 10% gain on the planar subset in CoFF. We rerun it with its released protocol on every benchmark. For the outdoor loop-closure regime, the strongest reported number is Pagor's 90.38% success on KITTI Hard in Pagor. We include Pagor as a geometry-plus-semantics reference there. On dynamic low-overlap pairs, no reported result was found on Lacuna for any method, so every dynamic number will come from our own runs.

The most dangerous baseline is mask-then-register. An off-the-shelf 2D segmenter removes all pedestrian and vehicle pixels, and CoFF is run on the remaining points with no other change. We also run Multi-Model 3D Registration on the same match pool as an explicit multi-body competitor. For fairness, all methods receive identical scan pairs, images and calibration, and the same fused-descriptor match pool wherever the method accepts external matches. Every learned component is retrained on the same training split with the same number of GPU-hours. $\lambda$ and $\tau$ are tuned on a held-out validation split only, and competitors receive a matching grid-search budget on their thresholds.

Beating CoFF and mask-then-register means higher registration recall on the dynamic benchmark in the strata where movers occupy a substantial share of the overlap. It also means registration recall on static 3DLoMatch within one point of CoFF's own rerun. The ablations isolate each ingredient. MAC's selector on our pool tests whether the score matters at all. Our score with free-space only tests whether images matter. Our score with image features only tests whether geometry matters. Removing the declaration of moved segments tests whether excusing motion matters.

Why not just…?

Why not just mask dynamic classes with a segmenter and run CoFF? This is the real threat, and we treat it as the primary comparison rather than a straw man. Masking loses in two situations. The first is when movers fill much of an already small overlap, so removing them starves the matcher. The second is when semantically "dynamic" objects are in fact parked and supply the only distinctive geometry. Split4D's authors argue that segmenter errors propagate into 3D. The benchmark varies the mover fraction and the parked-object fraction separately, so we can report exactly where masking holds up. If mask-then-register matches us across all strata, we will say so plainly. The contribution would then shrink to a verification step that helps only on parked-object scenes.

Why not just rely on MAC's many cliques, treating movers as outliers? A mover is not an outlier set in MAC's sense. It is a mutually consistent clique that passes every pairwise distance test. MAC's selection measures fit to the match set, not agreement with the rest of the scene. The failure audit will count how often MAC's chosen clique sits on a mover. The ablation that feeds our identical hypothesis pool through MAC's selector will show whether the selection score, rather than the pool, accounts for the gap.

Why not just use multi-body EM or Sequential RANSAC? Multi-Model 3D Registration already shows Sequential RANSAC collapsing, at 0.398 versus 0.970 IoU. The EM method, in turn, relies on dense scene flow and high overlap, which low-overlap pairs do not provide. We run it on our match pool anyway. If it recovers the background as reliably as our method does, that would mean clustering sparse matches is enough and dense verification is unnecessary.

Why not just add a geometric free-space check, with no images? Outdoors this may capture most of the gain, and we expect it to. The free-space-only ablation is designed to test exactly that. The images are there for indoor planar scenes, where several wrong hypotheses slide a wall along itself and remain fully free-space consistent. The prediction is that the image channel matters on the planar dynamic stratum and little elsewhere. If it adds nothing even there, the cross-modal component should be dropped.

Proposed Experiments

We build Dyn-3DLoMatch from 3DLoMatch pairs by rendering textured rigid object meshes into the RGB-D frames behind both fragments. Some objects move between the fragments and some stay put. We vary the share of the overlap they occupy from 0 to 60 percent and vary the fraction of static decoys. Registration recall under the standard 3DMatch thresholds is reported per stratum, along with the planar-enriched split. We also evaluate on KITTI Hard loop-closure pairs, where real traffic changes between visits. Using the ground-truth poses, we label moved points by free-space change detection and stratify results by moved-point fraction. We report success rate as defined by Pagor. For the static regression check, we report 3DLoMatch, 3DLoMatch_Planar, and ScanNet1500 rotation and translation accuracy against CoFF's and OG-UPCR's protocols. Runtime per pair is reported against MAC's tens of milliseconds.

The decisive result has three parts. Our method must beat mask-then-register CoFF in the high-mover and parked-object strata of both dynamic benchmarks, stay within one point of CoFF on static 3DLoMatch, and the failure audit must show that most of the recovered pairs were ones where the baseline had picked a mover's clique. If any of the three fails, we learn something specific. Either selection was not the bottleneck, or masking is sufficient, or the price of dense verification exceeds its benefit.