SIRA-PCR: Sim-to-Real Adaptation for 3D Point Cloud Registration

Suyi Chen, Hao Xu, Ru Li, Guanghui Liu, Chi-Wing Fu, Shuaicheng Liu

Abstract

Deep learning models for 3D point cloud registration often struggle with the domain gap between clean synthetic data and noisy real-world scans. SIRA-PCR addresses this by introducing a novel Sim-to-Real adaptation framework. It leverages a new, geometrically complex synthetic dataset, FlyingShapes, and a generative pipeline (SIRA) that mimics real-world sensor patterns. Central to SIRA is the Adaptive Re-sample Module (ARM), which adjusts point distributions while preserving underlying geometry. Experimental results demonstrate that SIRA-PCR achieves state-of-the-art performance on benchmarks like 3DMatch and 3DLoMatch, often surpassing models trained on manually annotated real-world data.

Introduction

3D point cloud registration is essential for aligning multiple 3D scans into a unified coordinate system, a critical task in robotics and computer vision. Despite the success of deep learning, these models require vast amounts of annotated data. Real-world datasets are expensive and difficult to label accurately, while existing synthetic datasets are often too simplistic, lacking the geometric complexity and sensor noise characteristic of real-world environments. This "domain gap" leads to poor generalization when models trained on synthetic data are deployed in the real world.

The SIRA-PCR Framework

The proposed framework tackles the domain gap through a two-stage process: generating complex synthetic scenes and adapting their point distributions to match real-world characteristics.

Overview of the SIRA-PCR pipeline, including FlyingShapes generation, Sim-to-Real Adaptation, and the registration process.

FlyingShapes Dataset

To bridge the gap in geometric complexity, the authors developed FlyingShapes. Built upon 3D-FRONT room layouts, the dataset is enriched with thousands of objects from ShapeNet. Objects are inserted using two strategies:

  • Physically-based insertion: Placing objects realistically on surfaces (e.g., mugs on tables).
  • Random insertion: Suspending objects in mid-air. This "clutter" forces the model to learn robust local features rather than relying on simple geometric primitives like walls.

Sim-to-Real Adaptation (SIRA)

Even with complex geometry, synthetic point clouds often exhibit "perfect" sampling that differs from real-world sensor output. SIRA uses a Generative Adversarial Network (GAN) to bridge this distribution gap. A multi-scale discriminator evaluates local patches, forcing the generator to produce realistic, fine-grained point distributions.

Adaptive Re-sample Module (ARM)

The core innovation of SIRA is the Adaptive Re-sample Module (ARM). Unlike standard GANs that might distort the underlying 3D structure, ARM learns to shift point positions by calculating a weighted sum of a point's neighbors.

The ARM structure and a 2D example illustrating how points are constrained within the convex hull of their neighbors to preserve geometry.

By restricting the movement of a point within the convex hull of its nearest neighbors, the module ensures that the global geometry remains intact while the local pattern (density and noise) is adapted to match real-world sensors. This constraint prevents the GAN from distorting the actual objects in the scene.

Experimental Results

The SIRA-PCR framework was evaluated using the GeoTransformer as the base registration model across several major benchmarks:

  • 3DMatch & 3DLoMatch: SIRA-PCR achieved state-of-the-art results with a Registration Recall (RR) of 94.1% on 3DMatch and 76.6% on the low-overlap 3DLoMatch.
  • Zero-Shot Generalization: When trained solely on synthetic data and tested on the real-world ETH outdoor dataset, SIRA-PCR improved RR by 1.9 percentage points over previous best methods, demonstrating exceptional robustness across different environments.
  • Inlier Ratio: The method significantly improved the percentage of correct point matches, enhancing the speed and reliability of the registration process.

Conclusion

SIRA-PCR demonstrates that high-quality synthetic data, combined with intelligent sim-to-real adaptation, can overcome the limitations of scarce real-world annotations. While the current focus is on indoor scenes, future work involves extending the FlyingShapes concept to outdoor environments and optimizing the computational overhead of the generative training pipeline.

Summary updated

Abstract

Point cloud registration is essential for many applications. However, existing real datasets require extremely tedious and costly annotations, yet may not provide accurate camera poses. For the synthetic datasets, they are mainly object-level, so the trained models may not generalize well to real scenes. We design SIRA-PCR, a new approach to 3D point cloud registration. First, we build a synthetic scene-level 3D registration dataset, specifically designed with physically-based and random strategies to arrange diverse objects. Second, we account for variations in different sensing mechanisms and layout placements, then formulate a sim-to-real adaptation framework with an adaptive re-sample module to simulate patterns in real point clouds. To our best knowledge, this is the first work that explores sim-to-real adaptation for point cloud registration. Extensive experiments show the SOTA performance of SIRA-PCR on widely-used indoor and out-door datasets. The code and dataset will be released on https://github.com/Chen-Suyi/SIRA_Pytorc.h
BibTeX
@inproceedings{chen2023sira,
  author = {Suyi Chen and Hao Xu and Ru Li and Guanghui Liu and Chi-Wing Fu and Shuaicheng Liu},
  title = {{SIRA-PCR: Sim-to-Real Adaptation for 3D Point Cloud Registration}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01324},
}
Figure