Learning to Localize in New Environments from Synthetic Training Data

Dominik Winkelbauer, Maximilian Denninger, Rudolph Triebel

TL;DR

This paper introduces ExReNet, a novel deep learning architecture for visual localization that learns to generalize to new environments using only synthetic training data. By incorporating hierarchical correlation layers and exploiting scale and uncertainty information, ExReNet significantly outperforms both classical and existing learning-based methods, especially in challenging scenarios with limited reference images.

What Problem Does This Paper Address?

Visual localization, the task of determining a robot's position and orientation within an environment, is a fundamental challenge in robotics. Traditional methods often rely on detailed 3D maps of the environment, which are expensive and time-consuming to create, or require retraining for each new scene. While learning-based approaches offer scene-agnostic capabilities, their generalization performance has historically lagged behind classical methods. This paper tackles the problem of achieving robust and generalizable visual localization without requiring detailed 3D models or scene-specific retraining, particularly in scenarios where only a few reference images are available.

Methods

The core of the proposed solution is a novel deep learning architecture called ExReNet, designed for relative pose estimation. ExReNet's key innovations include:

  • Hierarchical Correlation Layers: Instead of a single, high-resolution correlation layer, ExReNet employs a series of correlation layers. The initial layer performs an "extensive" correlation, while subsequent "guided" correlation layers leverage the matches found in previous layers to focus the search, reducing computational cost and improving accuracy. This is visualized in Figure 3.
  • Extended Regression Part: The network's regression module is made deeper and more complex than in prior work, which the authors found crucial for generalization.
  • Exploitation of Scale and Uncertainty: The approach explicitly predicts the scale of the relative translation between camera poses and estimates the uncertainty of these predictions. These pieces of information are then used to refine the triangulation step, making the final absolute pose estimation more robust.
  • Auxiliary Loss Function: To ensure that the correlation layers learn meaningful feature matches, an auxiliary loss function is introduced, which uses depth information to penalize incorrect matches.

The overall pipeline involves retrieving a few reference images, estimating relative poses using ExReNet, and then triangulating these relative poses to obtain the absolute pose of the query image.

Datasets

  • Training: The primary training dataset used is synthetic data generated using BlenderProc, specifically from the SUNCG dataset, which provides 3D models of furnished houses. Two types of synthetic training sets were used: one for densely covered scenes and another for sparsely covered scenes.
  • Evaluation: The approach was evaluated on the 7-Scenes dataset, a common benchmark for visual localization, and on a custom real-world dataset collected in a kitchen environment using an Intel RealSense D435 sensor.

Key Insights and Ideas

  • Synthetic Data for Generalization: Training a scene-agnostic model solely on synthetic data can lead to superior generalization to unseen real-world environments compared to training on specific scene datasets. This suggests that the diversity and controlled nature of synthetic data are highly beneficial for learning robust features.
  • Hierarchical Correlation is Key: The proposed hierarchical correlation layers are a significant improvement over single-layer correlation, offering a better trade-off between accuracy and computational efficiency. This approach allows the network to progressively refine its understanding of feature correspondences.
  • Scale and Uncertainty Matter: Explicitly predicting and utilizing translation scale and pose uncertainty information significantly enhances the robustness of the absolute pose estimation, particularly in challenging scenarios. This moves beyond simply relying on relative orientation.
  • Overfitting in Relative Pose Estimation: The paper demonstrates that even relative pose estimation methods can overfit to training trajectories, leading to poor performance on unseen data. Training on diverse synthetic data helps mitigate this overfitting.

Key Results

  • Outperforms Classical Methods: ExReNet, trained on synthetic data, outperforms the SIFT + 5-point algorithm on equally sized images.
  • Superior to Previous Learning-Based Methods: The proposed approach surpasses previous learning-based methods trained on different data and is also superior to most methods specifically trained on the 7-Scenes dataset.
  • Strong Performance with Few Reference Images: In scenarios with only four reference images (sparse maps), ExReNet significantly outperforms both classical and existing learning-based methods, achieving median translation errors as low as 0.61m and rotation errors around 7.74°.
  • Ablation Studies: Adding scale and uncertainty estimation to ExReNet further improves its performance, reducing translation errors. For instance, with both scale and uncertainty, the median translation error on the SUNCG dataset is 0.09m and rotation error is 2.72°.

Limitations and Future Work

  • Limitations: While the paper demonstrates strong generalization, the authors note that 3D structure-based methods still achieve slightly better accuracy on densely covered scenes like 7-Scenes. The performance on the ScanNet dataset was slightly worse than expected, which is attributed to the non-uniform sampling of images in ScanNet compared to the synthetic dataset.
  • Future Work: The paper suggests that further improvements could be made by exploring more advanced correlation layer designs and potentially incorporating semantic information into the localization process. The authors also highlight the potential for applying their method to real-time applications on mobile robots due to its efficiency.

Summary updated

Abstract

Most existing approaches for visual localization either need a detailed 3D model of the environment or, in the case of learning-based methods, must be retrained for each new scene. This can either be very expensive or simply impossible for large, unknown environments, for example in search-and-rescue scenarios. Although there are learning-based approaches that operate scene-agnostically, the generalization capability of these methods is still outperformed by classical approaches. In this paper, we present an approach that can generalize to new scenes by applying specific changes to the model architecture, including an extended regression part, the use of hierarchical correlation layers, and the exploitation of scale and uncertainty information. Our approach outperforms the 5-point algorithm using SIFT features on equally big images and additionally surpasses all previous learning-based approaches that were trained on different data. It is also superior to most of the approaches that were specifically trained on the respective scenes. We also evaluate our approach in a scenario with only very few reference images, showing that under such more realistic conditions our learning-based approach considerably exceeds both existing learning-based and classical methods.
BibTeX
@inproceedings{winkelbauer2021learning,
  author = {Dominik Winkelbauer and Maximilian Denninger and Rudolph Triebel},
  title = {{Learning to Localize in New Environments from Synthetic Training Data}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=asgqcb02dX},
}

Loading related content...

Figure