Registering two 3D scans means finding the rotation and translation that line one up with the other. When the scans overlap only a little, or when the scene repeats itself (a car roof next to a flat sidewalk, a row of identical lamp posts), matching on local shape alone produces mostly wrong correspondences. Semantic information fixes much of this. The strongest methods get it from image foundation models, the large networks such as DINOv2 that are pretrained on web photographs and produce general-purpose visual features. Those methods need a calibrated camera running next to the depth sensor at test time, and a LiDAR-only robot has none. We propose to use cameras only during training. From paired camera–LiDAR drives, a small LiDAR network learns to predict how semantically compatible two 3D points are, as judged by DINOv2. At test time this learned compatibility score, computed from the point cloud alone, reweights the candidate matches of a sensor-agnostic geometric pipeline.
The two sentences to carry away are these. We distill pairwise semantic compatibility, meaning whether two points look like the same kind of stuff, rather than per-point foundation features or class labels, from a 2D foundation model into a LiDAR-only network. We then use that compatibility as a soft semantic gate inside a training-free geometric registration pipeline. The bet is that this recovers most of the low-overlap gains of semantic registration with no camera and no segmentation labels at test time. A second bet is that, because the teacher is a generalist trained on natural images, the gate survives changes of LiDAR sensor better than a segmenter trained on one sensor's labels.
Where the gap is
Semantic-aware registration has clear evidence that semantics matter most exactly where geometry fails. ML-SemReg matches keypoints by the distribution of semantic labels in their neighborhoods and by ring-shaped semantic "fingerprints." On the low-overlap KITTI "hard" split, it raises the registration recall of the MAC robust estimator from 1.43% to 56.33%, and it lifts inlier ratios on standard KITTI from roughly 4.8% to over 20%. It needs a trained semantic segmenter, however. The continuous-kernel line of work, Zhang et al. and Ghaffari et al., multiplies a geometric kernel by a semantic kernel so that nearby but semantically incompatible points contribute nothing. This reduces KITTI translation error from 4.55% to 3.64%. That work supplies our scoring form, but it also assumes semantic class probabilities are available.
The foundation-model transfer methods split along the camera question. SemReg projects frozen DINOv2 features onto 3D points and uses them to guide attention and matching. It reaches 95.4% registration recall on 3DMatch and 77.4% on 3DLoMatch, but only with registered RGB at inference. ZeroReg chains Florence-2, SAMv2, CLIP and 2D matchers, and it also needs multi-view color. Two methods avoid cameras at test time. IAPReg renders six orthographic depth images and feeds them to a ViT, which works on objects (96.58% recall on ModelNet40) but suffers self-occlusion in open outdoor scenes and feeds depth renders to a photo-trained model. DBENet copies and freezes the point branch of a smaller multimodal teacher, reaching 93.8% on 3DMatch and 65.0% on 3DLoMatch. Its authors report that standard knowledge distillation from 2D to 3D was hard to optimize, and this negative result constrains our design. Finally, BUFFER-X generalizes zero-shot across 12 datasets and many LiDAR sensors using geometry alone. Its authors name low overlap and the lack of semantic cues as its open weakness. It is the host pipeline we want to fix.
The cross-domain transfer literature supplies the reason to prefer a generalist teacher. Gu et al. found that natural-image SAM beat medical-specific foundation models for zero-shot cross-modality registration. The specialist SSLSAM failed on unseen MRI sequences, and generalist SAM reached 87.0% Dice on breast contour alignment. The same paper shows these features are coarse: they capture outlines but miss thin structures. FM-OSD similarly finds frozen natural features beat in-domain pretraining for landmark detection. We take two lessons from this. First, generalist semantics may transfer across sensors better than specialist ones. Second, foundation features should decide which regions can match, while geometry decides where exactly. Our kernel product separates those two roles.
Core Claim
A LiDAR-only network trained to reproduce DINOv2's pairwise feature affinities between 3D points, within one scan and across scans, gives a label-free semantic compatibility score. Inserted multiplicatively into a geometric correspondence pipeline, this score substantially closes the low-overlap gap between geometry-only registration and camera-dependent semantic registration. It also degrades less than a supervised LiDAR segmenter when the LiDAR sensor changes.
Research Program
Training data come from sequences with calibrated cameras and known poses: KITTI odometry for outdoor scenes and the RGB-D fragments behind 3DMatch for indoor scenes. For each LiDAR point visible in an image, we look up the frozen DINOv2 patch feature $f_i$ at its projection. A point encoder, a sparse convolutional network over the raw point cloud, outputs a low-dimensional vector $g_i$ per point.
We do not regress $g_i$ onto $f_i$. That is the per-point route DBENet found brittle, and it would force a small 3D network to reproduce a 1024-dimensional image space. Instead we match relations. For an anchor point $i$ and a set $N_i$ of sampled partner points, drawn from the same scan and from a second scan aligned by ground-truth pose, we minimize
$$\mathcal{L}_i = \mathrm{KL}\big(\,\mathrm{softmax}_{j \in N_i}(\cos(f_i,f_j)/\tau_t)\;\big\|\;\mathrm{softmax}_{j \in N_i}(\cos(g_i,g_j)/\tau_s)\big),$$
where $\tau_t$ and $\tau_s$ are teacher and student temperatures. The loss only constrains who-resembles-whom, so the student can use any low-dimensional space that preserves DINOv2's similarity structure. The cross-scan partners are what make $g$ comparable between the source and target scans, which registration requires. Points outside the camera view receive no loss but still receive predictions at test time. This is a known limitation, and we measure it separately.
At test time no image is used. BUFFER-X, or FPFH as a weaker front end, produces candidate correspondences $(p,q)$ with geometric descriptor similarity $k_{\text{geo}}(p,q)$. Following the tensor-product kernel of Zhang et al., each candidate is weighted by
$$w(p,q) = k_{\text{geo}}(p,q)\cdot \sigma\big(\beta(\cos(g_p,g_q)-\delta)\big),$$
where $\sigma$ is the logistic function and $\beta,\delta$ set how sharply the gate closes. The gate is soft, so a misjudged point is downweighted rather than discarded. This addresses the segmentation-noise failure that ML-SemReg had to work around. The weights feed the robust solver (MAC or KISS-Matcher) as match priors. As a second use, we build a continuous analogue of ML-SemReg's neighborhood signature by averaging $g$ over concentric rings around each keypoint. This targets same-class ambiguity (two identical poles) without discrete labels.
Baselines
The naive reference point is BUFFER-X unchanged, with the same keypoints, descriptors and solver, run on every test split. Every gain we report is measured against it on identical inputs. The file gives no scalar recall for BUFFER-X on KITTI or 3DLoMatch (no reported result found on Lacuna for BUFFER-X on these splits), so we rerun it ourselves. We also run FPFH with MAC, whose KITTI hard recall ML-SemReg reports as 1.43%, so our runs can be checked against a published number.
The strongest label-based competitor is ML-SemReg driven by a LiDAR semantic segmenter trained with class labels on KITTI. It reports 56.33% registration recall on KITTI hard with MAC. It is the most dangerous rival because it uses the same geometric front end and adds semantics in exactly the regime we target, with human labels that our method lacks. It gets the same keypoints and solver as our method, and the segmenter receives the same training frames and GPU budget as our student. Near-parity with it on KITTI hard, without labels, counts as a success. Beating it on sensors it was not trained on is the main claim.
The strongest camera-free transfer competitor is DBENet, at 65.0% recall on 3DLoMatch and 93.8% on 3DMatch. On 3DLoMatch we compare a depth-only version of our method against these numbers and against a rerun under a matched budget. Exceeding 65.0% on 3DLoMatch without images would count as beating it. SemReg with RGB at test time (77.4% on 3DLoMatch) serves as an oracle upper bound rather than a competitor. We also evaluate our own gate with true projected DINOv2 features at test time, which measures how much the distillation step loses.
Three ablations isolate the key ingredient. The first replaces the relational loss with per-point regression onto $f_i$. The second drops the cross-scan partners, keeping only within-scan relations. The third replaces the soft gate with a hard threshold. If the per-point variant matches the relational one, our claim about why distillation works is wrong.
Why not just…?
Why not just train a LiDAR semantic segmenter and plug it into ML-SemReg? This is the real threat. Inside KITTI, where dense labels exist, it may win outright. The case for our method rests on two things. It needs no labels, and it may transfer across sensors, by analogy with the generalist-beats-specialist result of Gu et al.. That analogy is unproven for LiDAR. The zero-shot heterogeneous-LiDAR test decides it. If the segmenter also transfers across sensors, the contribution shrinks to label-free operation, and we will report it that way.
Why not just distill DINOv2 features point by point into the 3D encoder? DBENet reports that standard distillation between 2D grids and unordered point sets was hard to optimize, which is why it fell back to copying weights. Our first ablation runs per-point distillation directly under the same budget, so this question is answered empirically rather than argued.
Why not just render depth views and run the ViT at test time, as IAPReg does? Six fixed orthographic renders of an open street scene hide most of it. The method also pays several ViT forward passes per pair and feeds depth images to a photo-trained model. We run IAPReg-style rendering on KITTI as a baseline. If it matches our gate outdoors, the student is unnecessary.
Why not just rely on a better robust solver? ML-SemReg's result on KITTI hard, where MAC alone reaches 1.43% recall, shows that the solver is not the bottleneck when almost all candidate matches are wrong. BUFFER-X's authors say the same about low overlap.
Proposed Experiments
Training uses KITTI sequences 00–07 with camera features. Testing uses KITTI 08–10, the KITTI hard low-overlap split used by ML-SemReg, and 3DMatch and 3DLoMatch, where depth is the only test input. The decisive cross-domain test trains on KITTI only and evaluates zero-shot on the heterogeneous-LiDAR datasets in the BUFFER-X suite (WOD, KAIST, ETH, TIERS, MIT). These differ in beam count and mounting, and no labels or retraining are used for any method.
Metrics are registration recall, inlier ratio, rotation and translation error on successful pairs, and runtime added over BUFFER-X. The student is small and runs once per scan, so we expect the overhead to be milliseconds rather than the seconds needed by continuous-kernel methods.
Three outcomes would decide the proposal. The first is recall on KITTI hard relative to 1.43% (geometry-only) and 56.33% (label-based semantics). The second is 3DLoMatch recall relative to DBENet's 65.0%. The third is the size of the cross-sensor drop for our gate versus the supervised segmenter. A fourth check bins test correspondences by whether their points were ever visible to a training camera. This separates semantics actually lifted into LiDAR from interpolation near the camera frustum. If the gate helps only inside the frustum, the method has not learned to see without a camera.
References
- ML-SemReg: Boosting Point Cloud Registration with Multi-level Semantic Consistency
- A New Framework for Registration of Semantic Point Clouds from Stereo and RGB-D Cameras
- Progress in symmetry preserving robot perception and control through geometry and learning
- SemReg: Semantics Constrained Point Cloud Registration
- Zero-Shot Point Cloud Registration
- Partial Point Cloud Registration with Multi-view 2D Image Learning
- Boosting 3D Point Cloud Registration by Transferring Multi-modality Knowledge
- Towards Zero-Shot Point Cloud Registration Across Diverse Scales, Scenes, and Sensor Setups
- Are Vision Foundation Models Ready for Out-of-the-Box Medical Image Registration?
- FM-OSD: Foundation Model-Enabled One-Shot Detection of Anatomical Landmarks