TL;DR
Standard 3D point cloud registration pipelines break down when aligning a small, sparse point cloud (like a single LiDAR scan) to a much larger, denser scene (like a full 3D map). This paper introduces UPPNet, a hierarchical coarse-to-fine framework that handles unbalanced pairs by progressively narrowing the search space from candidate submaps to super-points, and finally to fine points. Alongside UPPNet, the authors propose KITTI-UPP, a benchmark dedicated to testing registration across severe scale and point-density discrepancies.
What Problem Does This Paper Address?
Most point cloud registration methods assume that the two input shapes share comparable spatial extents and point densities. In real-world robotic navigation, autonomous driving, and AR/VR, agents frequently need to register a partial, sparse local observation against an extensive pre-built map.
Under these unbalanced conditions:
- Pairwise feature matchers fail: Extensive maps contain repetitive geometric structures that act as hard negative distractors, causing high outlier ratios.
- Retrieval-based global localization methods struggle: They typically assume maps are partitioned into collections of overlapping scans rather than a single unified reconstruction, and they collapse under severe cross-domain density variations.
Methods
UPPNet tackles this problem using a shared KPConv-based U-Net architecture that extracts multi-scale features and registers unbalanced pairs hierarchically:
Input Pair (Query + Map)
│
▼
1. Submap Proposal & Matching
• Partition map into overlapping cubic regions scaled to the query size
• Aggregate super-point features with self/cross-attention and GeM pooling
• Retrieve top-k overlapping submaps via global descriptor similarity
│
▼
2. Super-Point Matching
• Solve optimal transport (Sinkhorn algorithm) on super-point similarity matrices
• Integrate local neighborhood geometric constraints
│
▼
3. Point Matching & Structured Outlier Filtering
• Refine matched super-points into local point patches
• Filter ambiguous matches using compatibility coherence (analyzing mutual spatial consistency)
• Compute final rigid transform (R, t) via RANSAC
- Submap Proposal & Matching: The map is divided into overlapping cubic submaps with edge lengths matching the query's bounding extent ($v = \max_i s_i$). After fusing query and map features using attention layers, Generalized Mean (GeM) pooling constructs a compact global descriptor per submap to select the top-$k$ candidates.
- Super-Point Matching: For candidate submaps, super-point features and neighboring point context are combined into a similarity matrix. The Sinkhorn algorithm resolves optimal transport to produce soft correspondence assignments.
- Structured Matching: To reject ambiguous matches, the authors compute a compatibility coherence matrix ($C = S S^\top$, where $S$ is the binary spatial consistency matrix). This counts how many consistent mutual neighbors two correspondences share, reliably pruning false positives caused by repetitive patterns.
- Point Matching: Selected super-point correspondences expand into local point patches for fine-grained matching and final pose estimation via RANSAC.
- End-to-End Training: The full pipeline is trained end-to-end using an optimal-transport loss combined across submap, super-point, and point scales ($L = L_s + \lambda_g L_g + \lambda_p L_p$).
Datasets
- KITTI-UPP (New Benchmark): Created from the KITTI Odometry dataset by aggregating sequential LiDAR sweeps into unified maps. It explicitly controls the spatial imbalance factor ($\rho_s$) using a
rangeparameter (number of aggregated frames) and the point density factor ($\rho_p$) using ahopparameter (frame skip). Query frames are temporally offset to avoid trivial scan self-matches. - ScanNet: Evaluates indoor unbalanced registration by aligning individual RGB-D frames against entire reconstructed room meshes.
- Standard Benchmarks (Balanced/Low-Overlap): Tested on standard KITTI, 3DMatch, and 3DLoMatch to verify that the model generalizes back to traditional pairwise registration setups.
Key Insights and Ideas
- Hierarchical spatial search beats global descriptor retrieval: Rather than summarizing an entire scene into a single global vector (like MinkLoc3D), proposing bounded submaps constrained by the physical size of the query preserves local distinctiveness and prevents dense point clouds from overwhelming sparse queries.
- Second-order geometric compatibility ($S S^\top$): Simple distance thresholding still admits outliers among repetitive structures. Measuring coherence—checking whether two candidate matches agree on their relationships with the rest of the match set—sharply differentiates true geometric overlap from repetitive distractors.
- Robustness across imbalance spectra: UPPNet does not require separate models for different map sizes; its submap decomposition inherently scales with query bounding dimensions.
Key Results
- KITTI-UPP: Under extreme imbalance ($\rho_s = 11.1\times$ larger spatial extent, $\rho_p = 11.7\times$ denser points), UPPNet achieves 82.5% Registration Recall (RR), outperforming the strongest baseline (CoFiNet at 62.9%) by 19.6%, and Predator (36.7%) and D3Feat (22.9%) by wider margins.
- ScanNet (Indoor): Achieves 71.0% RR compared to 50.0% from CoFiNet (+21.0% gain) and higher Inlier Ratio (64.1% vs 51.0%).
- Standard Benchmarks: Competitive with or superior to dedicated pairwise baselines:
- 3DMatch: 93.0% RR (vs. 89.3% CoFiNet, 89.0% Predator).
- 3DLoMatch: 69.0% RR (vs. 67.5% CoFiNet, 59.8% Predator).
- Runtime: Operates at 1.03s ($k=4$) to 2.28s ($k=16$) per pair on an Nvidia V100 GPU.
Limitations and Future Work
- Reliance on RANSAC: The final 6-DoF transformation estimation relies on classic RANSAC, which is non-differentiable and can add runtime overhead when candidate correspondences are high.
- Future Directions: Incorporating fully differentiable pose estimators (such as weighted Procrustes analysis) into the training pipeline to allow end-to-end backpropagation directly from pose error.
Summary updated