2D3D-DescNet: Jointly Learning 2D and 3D Local Feature Descriptors for Cross-Dimensional Matching

Shuting Chen, Yanfei Su, Baiqi Lai, Luwei Cai, Chengxi Hong, Li Li, Xiuliang Qiu, Hong Jia, Weiquan LiuRemote Sensing 2024 · DOI · OpenReview

TL;DR

This paper introduces 2D3D-DescNet, a novel network that jointly learns local feature descriptors for both 2D images and 3D point clouds. By bridging the gap between these two data modalities, it significantly improves cross-dimensional matching and retrieval tasks, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Establishing spatial relationships between 2D images and 3D point clouds is crucial for applications like augmented reality and robotics. However, a significant "domain gap" exists because 2D and 3D data are fundamentally different. Traditional and even deep learning-based feature descriptors are often specialized for either 2D or 3D, making direct comparison and matching challenging. This paper aims to create unified feature descriptors that are robust and comparable across both 2D and 3D domains, enabling seamless cross-dimensional matching.

Methods

The core of the proposed method is the 2D3D-DescNet, a multi-task network with three main components:

  • Feature Extractor: This module uses two separate encoders (one for 2D images, one for 3D point clouds) to learn initial feature representations.
  • Cross-Domain Image-Wise Feature Map Extractor: This is the most novel part. It employs a shared decoder to generate "image-wise feature maps" from both 2D and 3D features. Crucially, it integrates a Generative Adversarial Network (GAN). The discriminator in the GAN learns to distinguish the source of these feature maps (2D vs. 3D). This adversarial process forces the generator (and thus the feature extractor) to produce feature maps that are indistinguishable, effectively aligning the 2D and 3D feature representations.
  • Metric Network: A separate network takes the concatenated 2D and 3D feature descriptors and computes their similarity, outputting a probability of them being a match.

The network is optimized using a 2D-3D consistent loss function that includes:

  • Chamfer Loss: To ensure the 3D autoencoder reconstructs point clouds accurately.
  • Hard Triplet Margin Loss with SOS Regularization: To pull matching 2D and 3D descriptors closer in feature space while pushing non-matching ones further apart, bridging the domain gap.
  • Adversarial Loss (from GAN): To enforce the similarity of image-wise feature maps.
  • Cross-Entropy Loss: To train the metric network for accurate matching.

Datasets

The primary dataset used is 3DMatch, specifically focusing on its 2D-3D correspondences. The authors used 580,000 pairs for training and 20,000 for testing. They also validated their method on an outdoor 2D-3D dataset derived from a mobile LiDAR system, using 12,884 matching pairs for training and 1,000 for testing.

Key Insights and Ideas

  • GANs for Feature Alignment: The paper cleverly uses a GAN's adversarial training to force the network to learn 2D and 3D features that are not just similar in content but also in their "image-wise" representation, effectively minimizing the domain gap.
  • Joint Learning is Key: Instead of separate 2D and 3D descriptor learning, the joint approach with a shared decoder and adversarial loss proves highly effective in creating comparable descriptors.
  • Metric Network for Explicit Matching: While feature descriptors are learned, a dedicated metric network provides a direct measure of similarity, crucial for tasks like retrieval and matching.
  • Robustness Across Domains: The learned descriptors are shown to be robust, not only for retrieval but also for downstream tasks like 3D point cloud registration.

Key Results

  • State-of-the-Art Retrieval: 2D3D-DescNet achieved significant improvements in 2D-3D retrieval accuracy on the 3DMatch dataset, with TOP1 accuracy improved by over 12% compared to previous methods.
  • High Matching Performance: The metric network achieved a precision of 99.605% with a low False Positive Rate at 95% recall (FPR95) of 0.1891 on the 3DMatch dataset.
  • Effective 3D Registration: The learned 3D descriptors also demonstrated strong performance in 3D point cloud registration tasks, outperforming several traditional methods.
  • Descriptor Similarity: Visualizations (histograms) confirmed that the learned 2D and 3D descriptors are highly similar in their distributions.

Limitations and Future Work

  • Computational Cost: The paper notes that while the metric network provides excellent matching performance, its FCN architecture leads to high computational complexity, making it less practical for real-time applications compared to using retrieved descriptors.
  • Future Work: The authors plan to integrate their 2D and 3D local feature descriptors with keypoint detectors to develop a more complete solution for full 2D image and 3D point cloud matching.

Summary updated

Abstract

The cross-dimensional matching of 2D images and 3D point clouds is an effective method by which to establish the spatial relationship between 2D and 3D space, which has potential applications in remote sensing and artificial intelligence (AI). In this paper, we propose a novel multi-task network, 2D3D-DescNet, to learn 2D and 3D local feature descriptors jointly and perform cross-dimensional matching of 2D image patches and 3D point cloud volumes. The 2D3D-DescNet contains two branches with which to learn 2D and 3D feature descriptors, respectively, and utilizes a shared decoder to generate the feature maps of 2D image patches and 3D point cloud volumes. Specifically, the generative adversarial network (GAN) strategy is embedded to distinguish the source of the generated feature maps, thereby facilitating the use of the learned 2D and 3D local feature descriptors for cross-dimensional retrieval. Meanwhile, a metric network is embedded to compute the similarity between the learned 2D and 3D local feature descriptors. Finally, we construct a 2D-3D consistent loss function to optimize the 2D3D-DescNet. In this paper, the cross-dimensional matching of 2D images and 3D point clouds is explored with the small object of the 3Dmatch dataset. Experimental results demonstrate that the 2D and 3D local feature descriptors jointly learned by 2D3D-DescNet are similar. In addition, in terms of 2D and 3D cross-dimensional retrieval and matching between 2D image patches and 3D point cloud volumes, the proposed 2D3D-DescNet significantly outperforms the current state-of-the-art approaches based on jointly learning 2D and 3D feature descriptors; the cross-dimensional retrieval at TOP1 on the 3DMatch dataset is improved by over 12%.
BibTeX
@article{chen2024d,
  author = {Shuting Chen and Yanfei Su and Baiqi Lai and Luwei Cai and Chengxi Hong and Li Li and Xiuliang Qiu and Hong Jia and Weiquan Liu},
  title = {{2D3D-DescNet: Jointly Learning 2D and 3D Local Feature Descriptors for Cross-Dimensional Matching}},
  journal = {Remote Sensing},
  year = {2024},
  doi = {10.3390/rs16132493},
}
Figure