D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features

Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, Chiew-Lan Tai

Abstract

D3Feat is a novel deep learning framework that jointly tackles the two pillars of 3D point cloud registration: keypoint detection and feature description. By leveraging a density-invariant saliency score and a self-supervised loss guided by matching performance, it achieves state-of-the-art results while requiring significantly fewer points to align complex 3D scans.

Introduction

Point cloud registration—the process of aligning two 3D scans into a single coordinate system—is a foundational task for autonomous driving (SLAM) and 3D mapping. Traditionally, this process relies on identifying "keypoints" (distinctive locations) and generating "descriptors" (mathematical signatures of the local geometry).

While deep learning has revolutionized 3D descriptors, keypoint detection has often been handled through inefficient random sampling. Random sampling frequently selects points in non-salient regions, such as flat walls, leading to noisy matches and increased computational overhead. While 2D computer vision has moved toward "joint learning" of detection and description, adapting this to 3D is challenging due to the irregular structure and varying point density of 3D scans.

Methods

D3Feat proposes a dual-role fully convolutional network based on Kernel Point Convolution (KPConv). The architecture processes an unstructured point cloud in a single pass to produce a dense feature map.

The D3Feat pipeline: a dense feature extraction network outputs both descriptors and saliency scores for joint detection and description.

The framework introduces several core innovations:

  • Density-Invariant Saliency Score: To account for varying point densities in LiDAR scans, D3Feat compares a point’s feature response to the average of its local neighborhood, preventing the network from over-favoring sparse regions.
  • Joint Detection and Description: The network outputs a feature vector for every point. The detector acts as a selection mechanism, identifying the most "preeminent" channels and spatially local maxima from these features.
  • Self-Supervised Detector Loss: The detector is trained using the descriptor’s matching success. If a point pair matches correctly during training, the detector is encouraged to assign those points a higher saliency score.

Experimental Results

The authors evaluated D3Feat on three major benchmarks: 3DMatch (indoor RGB-D), KITTI (outdoor LiDAR), and ETH (generalization test).

Feature matching recall on the 3DMatch dataset across varying inlier ratio and distance thresholds.

Key Performance Metrics

  • State-of-the-Art Accuracy: On the 3DMatch dataset, D3Feat achieved a Feature Matching Recall of 95.8%, outperforming previous methods like FCGF (95.2%) and PerfectMatch (94.7%).
  • Outdoor Robustness: On the KITTI dataset, the model achieved a registration success rate of 99.81%, demonstrating high precision in large-scale environments.
  • Efficiency at Scale: A significant finding is that D3Feat remains stable even with extreme downsampling. Using only 250 well-chosen keypoints, it can outperform 5,000 randomly sampled points, maintaining a 93.3% recall on 3DMatch.
  • Inference Speed: D3Feat processes a fragment in approximately 0.13 seconds on a GTX1080, making it nearly 3x faster than sparse-convolution-based alternatives.

Discussion and Future Work

By coupling detection and description, D3Feat ensures the detector finds points that the descriptor is actually capable of matching. This "mutual agreement" leads to robust registration even in sparse or noisy conditions. Normalizing by local density proved to be a critical component for handling the inherent sparsity of 3D data.

While D3Feat shows strong generalization, it was slightly outperformed by PerfectMatch on the ETH dataset, likely due to that method's specific density representation. Future work aims to develop an end-to-end trainable registration pipeline that integrates geometric verification directly into the neural network, potentially removing the need for external RANSAC steps.

Summary updated

BibTeX
@inproceedings{bai2020d,
  author = {Xuyang Bai and Zixin Luo and Lei Zhou and Hongbo Fu and Long Quan and Chiew-Lan Tai},
  title = {{D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  doi = {10.1109/cvpr42600.2020.00639},
}
Figure