Self-Supervised Localisation between Range Sensors and Overhead Imagery

Tim Y. Tang, Daniele De Martini, Shangzhe Wu, Paul Newman

Abstract

Publicly available satellite imagery can be an ubiquitous, cheap, and powerful tool for vehicle localisation when a prior sensor map is unavailable. However, satellite images are not directly comparable to data from ground range sensors because of their starkly different modalities. We present a learned metric localisation method that not only handles the modality difference, but is cheap to train, learning in a self-supervised fashion without metrically accurate ground truth. By evaluating across multiple real-world datasets, we demonstrate the robustness and versatility of our method for various sensor configurations. We pay particular attention to the use of millimetre wave radar, which, owing to its complex interaction with the scene and its immunity to weather and lighting, makes for a compelling and valuable use case.

Introduction

For autonomous vehicles to navigate safely, precise localisation within a map is essential. Traditional methods rely on "prior maps"—highly detailed 3D scans created by specialized survey vehicles. However, the cost of building and maintaining these maps is a significant barrier to scalability.

Satellite imagery offers a ubiquitous and cost-effective alternative, but it presents a substantial modality gap. A top-down RGB satellite photo differs fundamentally from the sparse, noisy, ground-level scans produced by millimetre-wave radar or lidar. Previous deep learning approaches, such as RSL-Net, attempted to bridge this gap but required perfectly aligned "ground truth" training data, which is difficult and expensive to obtain. This paper introduces a self-supervised method that achieves metric localisation without the need for such metrically accurate labels.

Comparison of satellite imagery and corresponding radar scans, highlighting the significant modality gap between top-down RGB images and ground-level range data.

Methodology

The proposed pipeline translates between modalities using a two-stage self-supervised approach that disentangles scene appearance from geometric pose.

Self-Supervised Rotation Inference ($f_R$)

To resolve heading errors, the system employs a novel rotation inference network. The core innovation is a double-pass strategy: the network learns to identify which radar rotation matches a satellite image. By enforcing a cycle—where the network must also identify the original satellite image from a stack of rotated map images when given a correctly oriented radar scan—the system trains itself on orientation without requiring compass labels.

The self-supervised rotation inference pipeline ($f_R$) using a double-pass strategy to align modalities without ground-truth orientation labels.

Pose-Aware Separable Encoder Decoder (PASED)

At the heart of the image generation is PASED, which disentangles appearance (sensor-specific noise, shadows, and reflections) from pose (the sensor's geometric location). The model is first pre-trained on a single modality to learn appearance. Subsequently, the appearance and decoder modules are fixed, and a cross-modality pose encoder ($E^*_p$) is trained to find the translation between the satellite map and the live sensor scan.

Fourier Domain Correlation

After generating a synthetic radar image from the satellite map, the final $(x, y)$ offset is determined by maximizing the correlation between the synthetic and live images in the Fourier domain. This approach is computationally efficient and robust to noise.

Experimental Evaluation

The method was validated using three major real-world datasets: the Oxford Radar RobotCar Dataset, the MulRan Dataset, and the KITTI raw dataset. Satellite imagery was sourced from Google Maps and Bing Maps.

Key Results

  • Accuracy: The self-supervised method achieved mean errors between 3.4m and 6.0m for radar-to-satellite localisation. This performance is on par with state-of-the-art supervised methods like RSL-Net, despite the lack of expensive training labels.
  • Lidar Precision: Lidar-to-satellite localisation proved even more accurate, with mean errors as low as 1.5m.
  • Robustness: The system successfully tracked a vehicle's pose over a 1km trajectory using only radar and satellite imagery, requiring a GPS fix only for the initial frame.
  • Uncertainty Estimation: By applying known "fake" shifts to the map and testing the model's recovery capability, the system can estimate its own uncertainty, providing a critical safety feature for robotic applications.

Conclusion

This work demonstrates that self-supervised learning can effectively bridge the modality gap between ground-based range sensors and overhead imagery. By disentangling appearance from geometry, the system provides a scalable path for autonomous navigation in any environment covered by satellite imagery. Future work will focus on integrating this metric refinement with place recognition to solve the "Global Localisation" problem, enabling vehicles to find their position on a global map without any prior pose information.

Summary updated

Abstract

Publicly available satellite imagery can be an ubiquitous, cheap, and powerful tool for vehicle localisation when a prior sensor map is unavailable. However, satellite images are not directly comparable to data from ground range sensors because of their starkly different modalities. We present a learned metric localisation method that not only handles the modality difference, but is cheap to train, learning in a self-supervised fashion without metrically accurate ground truth. By evaluating across multiple real-world datasets, we demonstrate the robustness and versatility of our method for various sensor configurations. We pay particular attention to the use of millimetre wave radar, which, owing to its complex interaction with the scene and its immunity to weather and lighting, makes for a compelling and valuable use case.
BibTeX
@inproceedings{tang2020self,
  author = {Tim Y. Tang and Daniele De Martini and Shangzhe Wu and Paul Newman},
  title = {{Self-Supervised Localisation between Range Sensors and Overhead Imagery}},
  booktitle = {Robotics: Science and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=1BblUBXEuJs},
}
Figure