LHMap-loc: Cross-Modal Monocular Localization Using LiDAR Point Cloud Heat Map

Xinrui Wu, Jianbo Xu, Puyuan Hu, Guangming Wang, Hesheng Wang

Abstract

Precise localization is a cornerstone of autonomous driving. While LiDAR sensors provide rich 3D maps, their cost and data volume present significant challenges for mass-market vehicles. Monocular cameras offer a cost-effective alternative but struggle with cross-modal matching to 3D point clouds. This paper presents LHMap-loc, a high-performance pipeline that localizes a monocular camera within a LiDAR map by compressing point clouds into "LiDAR Heat Maps" (LHMap). By utilizing an optical flow-based pose regression network, LHMap-loc achieves state-of-the-art accuracy, maintains a 77 FPS processing speed, and reduces map storage requirements by 80%.

Introduction

The "where am I?" problem in autonomous driving is typically solved using expensive LiDAR sensors or high-definition (HD) maps. Monocular localization—matching a 2D image to a 3D map—is desirable for its low cost but faces two primary hurdles: the massive storage and computational overhead of raw LiDAR point clouds, and the inherent difficulty of cross-modal feature matching between 2D pixels and 3D points.

Overview of the LHMap-loc pipeline, showing the transition from raw LiDAR maps to compressed LHMaps and real-time pose regression.

LHMap-loc addresses these issues by identifying the most informative points in a LiDAR map for visual localization, creating a sparse yet effective representation that bridges the dimensionality gap.

Methodology

The LHMap-loc framework consists of an offline map generation stage and an online localization stage.

Offline LHMap Generation

Instead of keeping every point in a LiDAR map, the authors train a network to identify which points are most "useful" for visual localization based on visibility and distinctiveness. The network generates a "heat value" for each point. By selecting only the top 20% of points with the highest heat values, the system creates a LiDAR Heat Map (LHMap). This effectively compresses the original map by 80% while retaining the structural "skeleton" needed for matching.

Online Pose Regression

During operation, the system regresses the 6-Degree-of-Freedom (6-DoF) pose in real-time using a two-step approach:

  1. Projection and Flow: The LHMap is projected onto the current camera view. A deep network, inspired by optical flow architectures (PWCNet), calculates the displacement between the projected map and the actual RGB image.
  2. Spatial Attention: A weighting mechanism helps the network focus on reliable environmental features while ignoring noise or dynamic objects like other vehicles and pedestrians.
  3. End-to-End Learning: The network directly regresses the camera's position and orientation (quaternions).
Detailed architecture of the two-stage LHMap-loc pipeline, including offline generation and online pose regression modules.

Experiments and Results

The researchers evaluated their method on three major benchmarks: the KITTI Odometry Dataset, the Argoverse Dataset, and a custom SJTU Dataset collected in challenging real-world scenarios like lakeside paths and heavy foliage.

Key Results

  • Accuracy: On the KITTI dataset, LHMap-loc achieved a translation error of just 0.03 meters and a rotation error of 0.33° after three iterations.
  • Efficiency: The system processes at ~77 FPS (13.4ms per frame), making it more than fast enough for real-time autonomous navigation.
  • Reliability: While other methods showed failure rates on Argoverse, LHMap-loc maintained a 0% failure rate, demonstrating superior robustness.
  • Compression: The map size for KITTI sequence 00 was reduced from 511MB to 109MB without losing localization performance.

Discussion and Conclusion

LHMap-loc proves that "quality over quantity" is effective in mapping; 20% of carefully selected data can yield higher accuracy than 100% of raw data. Furthermore, using optical flow techniques to bridge the gap between 3D projections and 2D images is highly effective for cross-modal alignment. Future work will focus on handling long-term temporal changes (e.g., seasonal variations) and reducing the dependency on a rough initial pose for the regression process.

Summary updated

Abstract

Localization using a monocular camera in the pre-built LiDAR point cloud map has drawn increasing attention in the field of autonomous driving and mobile robotics. However, there are still many challenges (e.g. difficulties of map storage, poor localization robustness in large scenes) in accurately and efficiently implementing cross-modal localization. To solve these problems, a novel pipeline termed LHMap-loc is proposed, which achieves accurate and efficient monocular localization in LiDAR maps. Firstly, feature encoding is carried out on the original LiDAR point cloud map by generating offline heat point clouds, by which the size of the original LiDAR map is compressed. Then, an end-to-end online pose regression network is designed based on optical flow estimation and spatial attention to achieve real-time monocular visual localization in a pre-built map. In addition, a series of experiments have been conducted to prove the effectiveness of the proposed method. Our code is available at: https://github.com/IRMVLab/LHMap-loc.
BibTeX
@inproceedings{wu2024lhmap,
  author = {Xinrui Wu and Jianbo Xu and Puyuan Hu and Guangming Wang and Hesheng Wang},
  title = {{LHMap-loc: Cross-Modal Monocular Localization Using LiDAR Point Cloud Heat Map}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=Hw8qClZ8N2},
}
Figure