ModaLink: Unifying Modalities for Efficient Image-to-PointCloud Place Recognition

Weidong Xie, Lun Luo, Nanfei Ye, Yi Ren, Shaoyi Du, Minhang Wang, Jintao Xu, Rui Ai, Weihao Gu, Xieyuanli Chen

Introduction

Place recognition—the ability of an autonomous system to recognize a previously visited location—is a fundamental component for global localization and loop closure in robotics. While single-modality retrieval (e.g., image-to-image) is well-studied, cross-modal retrieval (matching a 2D camera image to a 3D point cloud map) is increasingly critical. This capability allows vehicles equipped with low-cost cameras to localize against high-fidelity LiDAR maps or enables heterogeneous robot teams to share spatial data.

The primary obstacle in cross-modal place recognition is the "modality gap": cameras capture photometric information, whereas LiDAR sensors capture geometric structure. Traditional approaches often attempt to bridge this gap by performing computationally intensive depth estimation on query images. However, these methods are often too slow for real-time edge deployment and require large amounts of labeled training data. ModaLink addresses these challenges by introducing a lightweight, real-time framework that achieves state-of-the-art accuracy at 30Hz.

Methodology

ModaLink shifts the paradigm by transforming 3D point clouds into a 2D format that mimics camera images, rather than the reverse. This approach significantly reduces computational overhead.

FoV Transformation and Depth Completion

The system first projects 3D LiDAR points onto a 2D plane using known camera intrinsics and extrinsics. To address the inherent sparsity of raw LiDAR projections, ModaLink employs a vertical interpolation technique to generate a "Dense Depth Image." This process aligns the geometric data into a format compatible with standard convolutional neural networks (CNNs), allowing for direct comparison with RGB query images.

Data processing pipeline: Point clouds are projected to 2D, cropped to the camera's Field of View, and processed into dense depth images.

NMF-Based Unsupervised Semantic Encoder

To extract distinctive and robust features, the framework incorporates a module based on Non-negative Matrix Factorization (NMF). NMF serves as an unsupervised clustering mechanism that identifies latent semantic structures—such as buildings or trees—within the feature maps. By focusing on these stable landmarks rather than transient pixel-level details, the model achieves higher discriminative power without the need for manual semantic labels or a dedicated segmentation network.

The shared-weight global descriptor encoder integrates a CNN backbone with an NMF module and NetVLAD layers.

Shared-Weight Network and Training

The architecture utilizes a shared ResNet34 backbone and NetVLAD layers to produce global descriptors for both query images and database depth images. During training, a triplet loss function is applied to ensure that descriptors from the same physical location are clustered together in the embedding space, while descriptors from different locations are pushed apart.

Experimental Results

ModaLink was evaluated on the KITTI benchmark and the large-scale HAOMO dataset, which includes complex urban environments and overpasses.

Accuracy and Efficiency

On the KITTI dataset (Sequence 00), ModaLink achieved a 98.0% Recall@1, outperforming the previous state-of-the-art (LEA-I2P-Rec at 92.6%). Crucially, the system maintains high efficiency, with the entire pipeline operating at approximately 33.45ms per frame (~30 FPS). This is an order of magnitude faster than competing methods where depth estimation alone can exceed 300ms.

Generalization

The model demonstrated strong zero-shot generalization when transferred from KITTI to the HAOMO dataset. After a brief refinement phase on the new environment, performance reached 70.9% Recall@1, proving its robustness across diverse geographical locations.

Conclusion

ModaLink provides a highly efficient solution for cross-modal place recognition by unifying 2D and 3D data through 2D projection and unsupervised semantic encoding. By bypassing the bottleneck of depth estimation, it enables real-time localization on resource-constrained autonomous platforms while setting new benchmarks for retrieval accuracy.

Summary updated

BibTeX
@inproceedings{xie2024modalink,
  author = {Weidong Xie and Lun Luo and Nanfei Ye and Yi Ren and Shaoyi Du and Minhang Wang and Jintao Xu and Rui Ai and Weihao Gu and Xieyuanli Chen},
  title = {{ModaLink: Unifying Modalities for Efficient Image-to-PointCloud Place Recognition}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801556},
}
Figure