Self-Supervised Point Set Local Descriptors for Point Cloud Registration

Yijun Yuan, Dorit Borrmann, Jiawei Hou, Yuexin Ma, Andreas Nüchter, Sören Schwertfeger

TL;DR

This paper introduces a self-supervised method for learning local descriptors in 3D point clouds, eliminating the need for expensive manual labeling. By using a single unlabeled point cloud and a differentiable "one-step" registration solver, the model learns to generate robust descriptors by simply trying to "undo" a random rotation applied to the data.

What Problem Does This Paper Address?

Point Cloud Registration (PCR)—the process of aligning two 3D scans into a single coordinate system—is a cornerstone of robotics, SLAM, and 3D reconstruction. Traditionally, this relies on "descriptors" (mathematical fingerprints of local geometry) to find correspondences between scans.

While deep learning has improved these descriptors, most state-of-the-art models are supervised, meaning they require massive amounts of human-labeled data or ground-truth poses to train. Labeling 3D correspondences is notoriously difficult and time-consuming. This paper tackles the "data hunger" problem by proposing a self-supervised framework that learns directly from raw, unlabeled point clouds.

Methods

The authors propose a clever training pipeline that treats registration as a pretext task:

  1. Data Augmentation: The system takes a single unlabeled point cloud ($PC_1$) and applies a known, random rotation ($R_{random}$) to create a second version ($PC_2$).
  2. Feature Extraction: Both point clouds pass through a shared "Descriptor" network (based on the 3DFeatNet architecture). This network maps local clusters of points into 32-dimensional descriptor vectors.
  3. The CF Module: Instead of using a standard triplet loss (which compares distances between points), the authors use their previously developed Full Connection Form Solution (CF) module. This is a differentiable layer that uses Singular Value Decomposition (SVD) to solve for the transformation between the two point sets in a single step.
  4. End-to-End Learning: Because the entire pipeline—from descriptor extraction to the SVD solver—is differentiable, the network can calculate the error between the predicted rotation and the actual random rotation. It then backpropagates this "rotation error" to update and improve the descriptors.

Datasets

The authors evaluated their approach using two major outdoor LiDAR benchmarks:

  • Oxford RobotCar Dataset: Used for both training (35 trajectories) and testing (5 trajectories). It contains 2D LiDAR scans accumulated into 3D clouds.
  • KITTI Dataset: Used exclusively for testing to demonstrate the model's ability to generalize to new environments.

Key Insights and Ideas

  • Registration as Supervision: The most elegant idea here is using the registration task itself as the loss function. If the descriptors are good, the SVD solver will correctly identify the random rotation. If they are bad, the rotation error will be high.
  • Efficiency in Training: Unlike supervised models that often require complex "triplet" sampling (finding matching and non-matching pairs), this model only needs one point cloud per iteration. The authors note it is significantly easier to train and converges faster.
  • Hand-crafted vs. Learned Detectors: The study shows that while the descriptor is learned, it performs best when paired with a good keypoint detector (like ISS). The learned descriptors are highly effective at describing "interesting" geometric features but are less distinctive when applied to random, featureless points.

Key Results

The self-supervised model achieved performance comparable to, and sometimes better than, fully supervised models:

  • Oxford Dataset: The proposed "ISS + Our Desc" achieved a 98.10% success rate with a Relative Rotation Error (RRE) of 1.01°, matching the performance of the supervised 3DFeatNet while being much simpler to train.
  • KITTI Dataset: On this "unseen" data, the model remained robust, achieving a 93.50% success rate and an RRE of 0.510°, outperforming several handcrafted baselines like FPFH and USC.
  • Training Speed: The model was saved at 72,500 iterations (equivalent to 20 epochs of 3DFeatNet), demonstrating high efficiency.

Limitations and Future Work

  • Random Point Distinctiveness: The model struggles to create unique descriptors for random points that aren't "keypoints." The authors suggest that the descriptor should always be used in conjunction with a keypoint detector rather than on a dense, uniform sampling.
  • Rotation Constraints: In the current experiments, the random rotations were limited to the Z-axis (gravity axis) to avoid unnecessary complexity. Expanding this to full 3D rotation is a logical next step.
  • Modality Expansion: The authors aim to test the method on indoor datasets and explore incorporating RGB (color) information into the descriptors to improve matching in visually rich environments.

Summary updated

Abstract

Descriptors play an important role in point cloud registration. The current state-of-the-art resorts to the high regression capability of deep learning. However, recent deep learning-based descriptors require different levels of annotation and selection of patches, which make the model hard to migrate to new scenarios. In this work, we learn local registration descriptors for point clouds in a self-supervised manner. In each iteration of the training, the input of the network is merely one unlabeled point cloud. Thus, the whole training requires no manual annotation and manual selection of patches. In addition, we propose to involve keypoint sampling into the pipeline, which further improves the performance of our model. Our experiments demonstrate the capability of our self-supervised local descriptor to achieve even better performance than the supervised model, while being easier to train and requiring no data labeling.
BibTeX
@inproceedings{yuan2021self,
  author = {Yijun Yuan and Dorit Borrmann and Jiawei Hou and Yuexin Ma and Andreas Nüchter and Sören Schwertfeger},
  title = {{Self-Supervised Point Set Local Descriptors for Point Cloud Registration}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=kAf3N3LqBLe},
}

Loading related content...

Figure