RKHS-BA: A Robust Correspondence-Free Multi-View Registration Framework with Semantic Point Clouds

Ray Zhang, Jingwei Song, Xiang Gao, Junzhe Wu, Tianyi Liu, Jinyuan Zhang, Ryan M. Eustice, Maani Ghaffari

TL;DR

This paper introduces RKHS-BA, a novel framework for multi-view bundle adjustment that uses continuous, semantic-rich representations in a Reproducing Kernel Hilbert Space (RKHS). By reformulating bundle adjustment as a correspondence-free pose graph problem, RKHS-BA achieves more robust and generalized convergence, particularly in challenging environments with noisy or sparse data.

What Problem Does This Paper Address?

Accurate 3D reconstruction and mapping are crucial for many robotics and computer vision applications, such as SLAM and SfM. Bundle Adjustment (BA) is a core technique for achieving this, but existing methods often struggle with perceptual degradation, such as low texture, illumination changes, or sensor noise. Feature-based BA methods rely on accurate feature correspondences, which can fail in challenging scenarios. Direct BA methods, while more robust to texture, can be sensitive to photometric inconsistencies and are not well-suited for sparse sensors like LiDAR. Furthermore, incorporating semantic information, which can significantly improve robustness, has been challenging due to issues with noise and generalization of learned semantic embeddings. This paper aims to address these limitations by developing a BA framework that is correspondence-free, robust to noise, and can effectively leverage both geometric and semantic information.

Methods

The core of RKHS-BA is its novel formulation of bundle adjustment using a pose graph and continuous functional representations in a Reproducing Kernel Hilbert Space (RKHS).

  • Continuous Functional Representation: Instead of explicit point-to-point correspondences, RKHS-BA represents each frame's observations (geometric and semantic) as a continuous function within an RKHS. This function implicitly encodes information about all points in the frame.
  • Correspondence-Free Pose Graph: The system constructs a pose graph where nodes are frames and edges represent relationships between them. The "alignment" between frames is measured by the inner product of their RKHS functions. This formulation eliminates the need for explicit feature matching.
  • Robust Optimization: The objective function, which maximizes the sum of inner products across the pose graph, is optimized using an Iteratively Reweighted Least Squares (IRLS) solver. This approach is designed to handle non-linearities and potential outliers more effectively than classical methods.
  • Global Rotation Initialization: To overcome local minima issues, especially with large initial rotations, RKHS-BA employs a novel strategy to sample the SO(3) manifold based on icosahedral symmetry, allowing for a robust initial pose guess.
  • Semantic Integration: The RKHS framework naturally accommodates various types of semantic information (e.g., color, pixel classes, learned features) alongside geometric data, allowing for a unified representation.

Datasets

The authors evaluate their method on several publicly available datasets:

  • Stanford Bunny Dataset [52]: Used for testing two-frame global rotation initialization and multi-point cloud registration with geometric data.
  • TartanAir Dataset [29]: Used for evaluating multi-point cloud registration with color and semantic information, as well as for sliding-window semantic bundle adjustment. This dataset provides photo-realistic simulations with ground truth depth and semantic labels.
  • SemanticKITTI Dataset [53]: Used for evaluating LiDAR global mapping capabilities.

The paper also mentions experiments on a self-collected dataset from a bipedal robot platform.

Key Insights and Ideas

  • Correspondence-Free is Key: Eliminating the need for explicit point-to-point correspondences is a significant step towards robustness, especially in scenes with sparse overlap or challenging features.
  • RKHS as a Unifying Space: The RKHS provides a powerful mathematical framework to represent and fuse diverse data types (geometric and semantic) into a single, coherent representation for optimization.
  • Semantic Information Enhances Robustness: Incorporating semantic data, even when noisy, demonstrably improves registration accuracy and convergence, particularly in challenging scenarios.
  • Global Rotation Initialization Solves a Critical Problem: The proposed icosahedral sampling method for initializing rotations is crucial for avoiding local minima in real-world applications where initial poses can be highly uncertain.
  • Generalized Convergence: The RKHS norm convergence implies point-wise convergence, offering a more generalized and potentially stronger convergence criterion than traditional methods.

Key Results

  • Robust Registration: RKHS-BA demonstrates superior performance in two-frame registration under significant outlier ratios (up to 50%) and cropping, outperforming methods like RANSAC and FGR, especially with large initial rotations (90° and 180°).
  • Multi-Point Cloud Registration: In multi-view registration experiments, RKHS-BA achieves lower pose errors compared to baselines like JRMPC, particularly when dealing with outliers.
  • Sliding Window Semantic BA: On the TartanAir dataset, RKHS-BA with semantic information achieves lower mean Absolute Translation Error (ATE) than other direct and feature-based methods, showing robustness in challenging environments with dynamic weather and textureless areas.
  • LiDAR Global Mapping: RKHS-BA shows improved performance over existing LiDAR global mapping methods, achieving lower translation and rotational errors, especially when using semantic information.
  • Semantic Noise Tolerance: While extreme semantic noise can impact performance, the method shows a degree of resilience to moderate noise levels.

Limitations and Future Work

  • Computational Complexity: The current implementation has a time complexity of O(MN^2) for M edges and N points, which can be computationally intensive.
  • Lengthscale Initialization: The initial lengthscale choice can affect gradient calculation, requiring a sufficiently large initial value which may increase computation time.
  • Semantic Noise Sensitivity: While robust to some noise, extreme semantic noise can still degrade performance.

Future work includes:

  • More efficient implementations using voxel hashing on GPUs for parallel processing.
  • Integrating dense differentiable mapping for photorealistic rendering.
  • Further exploration of semantic information and its impact on robustness.

Summary updated

Abstract

This work reports a novel multi-frame Bundle Adjustment (BA) framework called RKHS-BA. It uses continuous landmark representations that encode RGB-D/LiDAR and semantic observations in a Reproducing Kernel Hilbert Space (RKHS). With a correspondence-free pose graph formulation, the proposed system constructs a loss function that achieves more generalized convergence than classical point-wise convergence. We demonstrate its applications in multi-view point cloud registration, sliding-window odometry, and global LiDAR mapping on simulated and real data. It shows highly robust pose estimations in extremely noisy scenes and exhibits strong generalization with various types of semantic inputs. The open source implementation is released in https://github.com/UMich-CURLY/RKHS_BA.
BibTeX
@article{zhang2025rkhs,
  author = {Ray Zhang and Jingwei Song and Xiang Gao and Junzhe Wu and Tianyi Liu and Jinyuan Zhang and Ryan M. Eustice and Maani Ghaffari},
  title = {{RKHS-BA: A Robust Correspondence-Free Multi-View Registration Framework with Semantic Point Clouds}},
  journal = {IEEE TPAMI},
  year = {2025},
  doi = {10.1109/tpami.2025.3593521},
}
Figure