Self-supervised learning for using overhead imagery as maps in outdoor range sensor localization

Tim Y. Tang, Daniele De Martini, Shangzhe Wu, Paul Newman

TL;DR

This paper introduces a self-supervised learning method for robots to localize themselves using overhead satellite imagery as a map, even when the robot's onboard sensors (like radar or lidar) are of a different modality. This approach bypasses the need for expensive, precisely aligned ground truth data during training, making it more accessible and practical for real-world applications.

What Problem Does This Paper Address?

Accurate robot localization is crucial for autonomous navigation. Traditional methods rely on pre-built maps created using the same sensor type as the robot's onboard sensors. However, obtaining such maps can be inconvenient or costly, especially for outdoor environments. Publicly available overhead imagery, like satellite photos, offers a ubiquitous and cheap alternative. The main challenge is the "modality gap": satellite images are 2D aerial views, while range sensors like radar and lidar provide sparse 3D point clouds or intensity maps. Directly comparing these is difficult. Existing methods that bridge this gap often require pixel-wise aligned ground truth data for training, which is expensive to acquire and limits their applicability. This paper addresses this by developing a self-supervised method to learn cross-modality localization without needing this precise ground truth.

Methods

The core of the proposed method is a self-supervised learning framework that generates a synthetic range sensor image that matches the appearance and scene of a live range sensor reading, but is aligned with a satellite map. This is achieved in three main steps:

  1. Rotation Inference: The system first estimates the rotational offset between the satellite map and the live range sensor data. This is done by creating a stack of rotated versions of the live sensor image and using a network to predict which rotation best aligns with the satellite map. Crucially, this is learned self-supervisedly by having the network predict the alignment from multiple perspectives.
  2. Image Generation: A novel architecture called PASED (Pose-Aware Self-supervised Encoder-Decoder) is introduced. It's trained in two stages: first, an intra-modality pre-training step learns to generate shifted versions of images within the same modality. Then, a cross-modality step uses this pre-trained knowledge to learn to generate a synthetic range sensor image that aligns with the satellite map, conditioned on the live sensor data. This ensures the generated image contains relevant scene information.
  3. Pose Estimation: Finally, the system projects the real and synthetic range sensor images into a joint embedding space. The translation offset is then estimated by maximizing the correlation between these embeddings. This correlation maximization is a parameter-free process that is differentiable, allowing gradients to flow back to the learned components.

Datasets

The authors evaluate their method on several public datasets:

  • Oxford Radar RobotCar Dataset: Contains radar and lidar data with GPS/INS ground truth.
  • MulRan Dataset: Similar to RobotCar, providing radar and GPS/INS data.
  • KITTI Dataset: Includes lidar data and GPS ground truth.

Satellite images for these datasets were queried using Google Maps Platform and Bing Maps Platform.

Key Insights and Ideas

  • Self-Supervision is Key: The most significant insight is the successful application of self-supervised learning to cross-modality localization. This removes the bottleneck of acquiring expensive, precisely aligned ground truth data, making the approach much more practical.
  • Modality Gap Bridging: The method effectively bridges the modality gap by generating synthetic images that translate between different sensor types, rather than relying on hand-crafted features or direct regression.
  • PASED Architecture: The proposed PASED architecture, with its two-stage pre-training and cross-modality learning, is a novel contribution for learning cross-modal image generation for localization.
  • Introspection for Robustness: The system incorporates an "introspection" mechanism to detect when its localization confidence is low, allowing it to fall back to odometry or re-evaluate its pose, enhancing robustness.
  • Handling Large Initial Offsets: The method can be adapted to handle initial pose offsets larger than those seen during training by using input augmentation techniques.

Key Results

  • The proposed self-supervised method achieves localization errors on par with a state-of-the-art supervised approach (RSL-Net) on various datasets and sensor configurations (radar/lidar against satellite imagery, and radar against lidar maps).
  • For radar localization against satellite imagery on the RobotCar dataset, the mean translation error was around 3.44m and 5.40m for x and y respectively, with a heading error of 3.03°.
  • For lidar localization against satellite imagery on the RobotCar dataset, the mean translation error was around 1.54m and 1.85m for x and y, with a heading error of 2.29°.
  • The method demonstrates robustness across different environments (urban, residential, campus, highway) and sensor configurations.
  • The self-supervised approach significantly reduces the need for expensive ground truth data.

Limitations and Future Work

  • Coarse Initial Estimate: The method still requires a coarse initial pose estimate, typically provided by a place recognition system or GPS. It does not perform global localization from scratch.
  • Computational Cost: While self-supervised, the method involves multiple network passes and image generation steps, which can increase computational cost, especially for handling large initial offsets.
  • Future Work: The authors suggest extending the work to include place recognition for range sensors within large satellite maps as a prior step to metric localization. They also mention that modality transfer (e.g., training with radar data and testing with lidar) is possible and can be improved with domain adaptation techniques.

Summary updated

BibTeX
@article{tang2021self,
  author = {Tim Y. Tang and Daniele De Martini and Shangzhe Wu and Paul Newman},
  title = {{Self-supervised learning for using overhead imagery as maps in outdoor range sensor localization}},
  journal = {Int. J. Robotics Res.},
  year = {2021},
  doi = {10.1177/02783649211045736},
}
Figure