Self-Supervised Localisation between Range Sensors and Overhead Imagery

Tim Y. Tang, Daniele De Martini, Shangzhe Wu, Paul Newman

Abstract

Reliable localization is a cornerstone of autonomous vehicle technology. While traditional methods rely on labor-intensive, sensor-specific maps, overhead satellite imagery offers a globally available alternative. However, the significant "modality gap" between RGB satellite photos and ground-based radar or lidar scans typically necessitates high-precision ground truth for supervised training. This paper introduces a self-supervised framework that bridges this gap through synthetic image generation and internal geometric consistency. By disentangling scene appearance from pose, the proposed system achieves localization accuracy comparable to state-of-the-art supervised methods without requiring expensive, high-end GPS/INS data for training.

Introduction

Traditionally, robots localize by comparing live sensor data to a pre-built map generated with the same sensor (e.g., matching lidar to a lidar map). Building and maintaining these maps is a significant bottleneck for scaling autonomous systems. Publicly available satellite imagery provides a "free" global map, but matching a ground-based point cloud to an overhead photo is difficult due to the drastic difference in perspective and data format.

Comparison of overhead satellite imagery and ground-based range sensor data (radar and lidar), highlighting the modality gap.

Previous attempts to bridge this gap required metrically accurate ground truth (sub-meter and sub-degree precision) to train deep learning models. Collecting such data requires high-end positioning systems and complex post-processing. This research addresses this limitation by proposing a self-supervised pipeline that learns to localize by generating synthetic images and enforcing internal consistency.

Methodology

The proposed pipeline consists of three main components designed to handle unknown headings and translation offsets without external labels.

Self-Supervised Rotation Inference

To handle unknown headings, the system employs a "double-pass" strategy. It first identifies the best-aligned image from a stack of rotated live scans. This result is then used to "reconstruct" the original map from a stack of rotated map images. By enforcing consistency between these two passes using an $L_1$ loss, the network learns rotation without ever seeing a "correct" label.

The proposed self-supervised pipeline for cross-modal localization using a double-pass consistency strategy and the PASED architecture.

Pose-Aware Separable Encoder Decoder (PASED)

This architecture is the core of translation estimation. It disentangles appearance (the sensor-specific style) from pose (the geometric location of objects). A "known shift" trick is used for supervision: the network is forced to predict a manually injected translation offset. Since this offset is known to the researchers but not the model, it serves as a built-in supervisor for learning spatial relationships.

Metric Localization

Once a synthetic radar or lidar image is generated from the satellite map, the system compares it to the live sensor data. A correlation-based approach in the Fourier domain is used to find the precise $(x, y)$ offset at 10Hz, enabling real-time operation.

Experimental Results

The study validates the approach across several diverse, real-world datasets, including the Oxford Radar RobotCar Dataset, the MulRan Dataset (covering campus and highway scenes), and the KITTI benchmark for lidar.

The self-supervised method performs on par with RSL-Net, the previous state-of-the-art supervised method. Key performance metrics include:

  • Radar vs. Satellite: On the RobotCar dataset, the system achieved a mean metric error of approximately 3.44m (x) and 5.40m (y).
  • Lidar vs. Satellite: Performance was even sharper, with errors around 1.54m (x) and 1.85m (y).
  • Cross-Modality Versatility: The method successfully localized radar data against prior lidar maps, outperforming naive image-to-image translation methods like CycleGAN.

Discussion and Conclusion

A key insight of this work is that disentanglement is essential for cross-modal tasks. By separating sensor style from scene geometry, the model learns to translate between modalities without needing perfectly aligned pairs. Furthermore, the use of internal consistency as supervision proves that external GPS is not strictly necessary if the robot's internal logic can be proven inconsistent when errors occur.

While the system excels at tracking, it currently requires a coarse initial pose and can experience drift over long trajectories without loop closures. Future work will focus on integrating place recognition directly into this framework to create a truly end-to-end global localization system capable of solving the "kidnapped robot" problem.

Summary updated

Abstract

Publicly available satellite imagery can be an ubiquitous, cheap, and powerful tool for vehicle localisation when a prior sensor map is unavailable. However, satellite images are not directly comparable to data from ground range sensors because of their starkly different modalities. We present a learned metric localisation method that not only handles the modality difference, but is cheap to train, learning in a self-supervised fashion without metrically accurate ground truth. By evaluating across multiple real-world datasets, we demonstrate the robustness and versatility of our method for various sensor configurations. We pay particular attention to the use of millimetre wave radar, which, owing to its complex interaction with the scene and its immunity to weather and lighting, makes for a compelling and valuable use case.
BibTeX
@inproceedings{tang2020self,
  author = {Tim Y. Tang and Daniele De Martini and Shangzhe Wu and Paul Newman},
  title = {{Self-Supervised Localisation between Range Sensors and Overhead Imagery}},
  booktitle = {Robotics: Science and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=1BblUBXEuJs},
}
Figure