Learning to Localize Using a LiDAR Intensity Map

Ioan Andrei Bârsan, Shenlong Wang, Andrei Pokrovsky, Raquel Urtasun

Abstract

In this paper we propose a real-time, calibration-agnostic and effective localization system for self-driving cars. Our method learns to embed the online LiDAR sweeps and intensity map into a joint deep embedding space. Localization is then conducted through an efficient convolutional matching between the embeddings. Our full system can operate in real-time at 15Hz while achieving centimeter level accuracy across different LiDAR sensors and environments. Our experiments illustrate the performance of the proposed approach over a large-scale dataset consisting of over 4000km of driving.

Introduction

Precise localization is a fundamental requirement for autonomous vehicles, typically necessitating centimeter-level accuracy. Traditional localization techniques generally rely on geometric matching (e.g., Iterative Closest Point) or intensity-based matching. Geometric methods often struggle in environments with sparse features, such as highways or tunnels. Intensity-based methods, while more descriptive, are highly sensitive to sensor calibration and environmental changes.

LiDAR intensity map and corresponding local crops used for matching.

This work introduces a system that is calibration-agnostic and robust to environmental variations by leveraging deep learning to process LiDAR intensity data. By embedding both online sweeps and pre-built maps into a shared feature space, the system learns to focus on stable road features while ignoring sensor-specific noise.

Methodology

The core of the system is a Deep Matching Network that processes online LiDAR data and a high-definition (HD) intensity map.

Deep Matching and Joint Embedding

The architecture utilizes two deep neural networks to transform noisy online LiDAR sweeps and clean HD map data into a joint embedding space. This transformation allows the model to extract reliable features for matching, effectively bypassing the need for manual sensor calibration.

Overview of the proposed deep localization system architecture.

Efficient Inference

To achieve real-time performance, the system employs several key techniques:

  • FFT-based Matching: The search for the optimal vehicle pose (translation and rotation) is performed in the frequency domain using Fast Fourier Transforms (FFT). This turns a computationally expensive exhaustive search into a highly efficient operation.
  • Recursive Bayesian Inference: The system maintains a probability distribution (belief) over the vehicle's state, updated over time by integrating motion models (odometry) and GPS data with the LiDAR-based observations.
  • Sub-pixel Accuracy: A soft-argmax operation is applied to the matching scores to achieve centimeter-level precision beyond the discrete grid resolution.

Experimental Evaluation

The system was evaluated on a massive dataset comprising over 4,000 km of driving across North American cities.

Datasets

  • Highway-LidarA: 3,000+ km of data used for primary training and validation.
  • Misc-LidarB: A 200 km transfer set collected with a different LiDAR model to test hardware generalization.

Key Results

  • Precision: The system achieved a median error of 6–7 cm in both lateral and longitudinal directions.
  • Robustness: On highway segments, the proposed method maintained a 0% failure rate (error > 1m), significantly outperforming geometric ICP methods which failed over 7% of the time.
  • Generalization: Without retraining, the system maintained high accuracy (~11 cm median error) when deployed on entirely different LiDAR hardware.
  • Speed: The entire pipeline operates at 15Hz, meeting the real-time constraints of autonomous driving.

Conclusion

This research demonstrates that localization can be framed as a learning problem to overcome the limitations of manual calibration and geometric sparsity. By utilizing deep embeddings and efficient frequency-domain matching, the proposed system provides a robust, real-time solution for high-precision autonomous vehicle localization across diverse environments and hardware platforms.

Summary updated

Abstract

In this paper we propose a real-time, calibration-agnostic and effective localization system for self-driving cars. Our method learns to embed the online LiDAR sweeps and intensity map into a joint deep embedding space. Localization is then conducted through an efficient convolutional matching between the embeddings. Our full system can operate in real-time at 15Hz while achieving centimeter level accuracy across different LiDAR sensors and environments. Our experiments illustrate the performance of the proposed approach over a large-scale dataset consisting of over 4000km of driving.
BibTeX
@inproceedings{barsan2018learning,
  author = {Ioan Andrei Bârsan and Shenlong Wang and Andrei Pokrovsky and Raquel Urtasun},
  title = {{Learning to Localize Using a LiDAR Intensity Map}},
  booktitle = {Conference on Robot Learning},
  year = {2018},
  url = {https://openreview.net/forum?id=d_FCcPw2Kl9j},
}
Figure