CMSG: Cross-Media Semantic-Graph Feature Matching Algorithm for Autonomous Vehicle Relocalization

Shuhang Tan, Hengyu Liu, Zhiling WangarXiv 2023 · arXiv

Abstract

Relocalization is the basis of map-based localization algorithms. Camera and LiDAR map-based methods are pervasive due to their robustness under different scenarios. Generally, mapping and localization using the same sensor have better accuracy since matching features between the same type of data is easier. However, due to the camera's lack of 3D information and the high cost of LiDAR, cross-media methods are developing which combine live image data and LiDAR maps. Although matching features between different media is challenging, we believe cross-media is the tendency for AV relocalization since its low cost and accuracy can be comparable to same-sensor-based methods. In this paper, we propose CMSG, a novel cross-media algorithm for AV relocalization tasks. Semantic features are utilized for better interpretation of the correlation between point clouds and image features. Furthermore, abstracted semantic graph nodes are introduced, and a graph network architecture is integrated to better extract the similarity of semantic features. Validation experiments are conducted on the KITTI odometry dataset. Our results show that CMSG can achieve comparable or even better accuracy compared to current single-sensor-based methods at a speed of 25 FPS on an NVIDIA 1080 Ti GPU.

Introduction

High-precision relocalization is a fundamental requirement for autonomous vehicle (AV) navigation. While Global Positioning Systems (GPS) provide a baseline, their accuracy is often insufficient for the centimeter-level precision required for safe driving. Map-based localization, which matches live sensor data against high-definition (HD) maps, offers a more robust solution. Traditionally, this process involves matching homogeneous data, such as live LiDAR scans against a pre-built LiDAR map. However, the high cost of LiDAR sensors limits their widespread adoption in mass-market vehicles.

Cameras offer a cost-effective alternative, but matching 2D image data to 3D LiDAR point clouds presents a significant "modality gap." The CMSG (Cross-Media Semantic-Graph) algorithm addresses this challenge by utilizing semantic information as a common language to bridge the gap between disparate sensor types.

The CMSG Framework

The core innovation of CMSG lies in its use of semantic graphs to represent the environment. By abstracting raw sensor data into semantic nodes and their spatial relationships, the system becomes invariant to the specific sensor modality.

The CMSG pipeline architecture, illustrating the dual encoding of point clouds and images, semantic graph construction, and attention-based fusion.

The pipeline consists of several key stages:

  1. Dual Encoding: The system employs a ResNet-34 backbone for image feature extraction and a combination of So-net and Pointnet++ for 3D point cloud processing.
  2. Semantic Abstraction: Semantic segmentation (using Deeplabv3+ for images and Rangenet++ for LiDAR) identifies objects like cars, buildings, and vegetation, providing a shared conceptual framework.
  3. Graph Construction: Points and pixels are clustered into semantic nodes. For LiDAR, a distance-adaptive clustering algorithm is used, while image nodes are clustered based on central coordinates.
  4. Graph Similarity Network: An EdgeConv structure encodes the topological relationships between nodes, allowing the model to match the spatial arrangement of objects rather than just raw features.
  5. Attention-Based Fusion: An attention mechanism weighs the importance of various features before a multilayer perceptron (MLP) performs the final similarity classification.

Experimental Results

The CMSG algorithm was validated using the KITTI odometry dataset, specifically sequences 00-10. Positive samples were defined as pairs within a 2-meter radius, while negative samples were separated by more than 20 meters.

Comparison of F1 scores across various KITTI sequences, showing CMSG's performance relative to state-of-the-art methods.

As shown in the results, CMSG achieved an average F1 score of 0.816. While this is slightly below the LiDAR-only Scan Context method (0.914), it significantly outperforms other single-sensor benchmarks such as M2DP (0.719) and PointNetVLAD (0.627). Notably, in sequence 08, CMSG demonstrated superior robustness, outperforming all other tested methods.

Beyond accuracy, CMSG is designed for real-time deployment. It achieves a processing speed of 25 FPS on an NVIDIA 1080 Ti GPU. The model's efficiency is further enhanced by an offline feature extraction process for the LiDAR map, minimizing the real-time computational burden on the vehicle.

Conclusion

CMSG demonstrates that cross-media relocalization can achieve accuracy comparable to single-sensor methods while maintaining the cost advantages of camera-based systems. By leveraging semantic graphs, the algorithm effectively bridges the modality gap. Future work will focus on addressing the field-of-view asymmetry between panoramic LiDAR and narrow-angle cameras to further enhance matching precision.

Summary updated

Abstract

Relocalization is the basis of map-based localization algorithms. Camera and LiDAR map-based methods are pervasive since their robustness under different scenarios. Generally, mapping and localization using same sensor has better accuracy since matching feature between same type of data is easier. However, due to camera's lack of 3D information and the high cost of LiDAR, cross-media methods are developing, which combined live image data and Lidar map. Although matching features between different medias is challenging, we believe cross-media is the tendency for AV relocalization since its low-cost and the accuracy can be comparable to the same-sensor based methods. In this paper, we propose CMSG, a novel cross-media algorithm for AV relocalization task. Semantic features are utilized for better interpretating the correlation between point clouds and image features. What's more, abstracted semantic graph nodes are introduced, and a graph network architecture is integrated to better extract the similarity of semantic feature. Validation experiments are conducted on the KITTI odometry dataset. Our results show that CMSG can have comparable or even better accuracy compared to current single-sensor based methods at a speed of 25 FPS on NVIDIA 1080 Ti GPU. Related code can be found in https://github.com/Clouds1997/CMSG
BibTeX
@misc{tan2023cmsg,
  author = {Shuhang Tan and Hengyu Liu and Zhiling Wang},
  title = {{CMSG: Cross-Media Semantic-Graph Feature Matching Algorithm for Autonomous Vehicle Relocalization}},
  year = {2023},
  eprint = {2305.08318},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.08318},
}
Figure