StickyLocalization: Robust End-to-End Relocalization on Point Clouds Using Graph Neural Networks

Kai Fischer, Martin Simon, Stefan Milz, Patrick MäderWACV 2022

TL;DR

This paper introduces "StickyLocalization," a novel end-to-end deep learning approach for robust 3D point cloud relocalization. By leveraging Graph Neural Networks (GNNs) for context aggregation and a new distance-based matching loss, it achieves state-of-the-art performance in both accuracy and speed, even with challenging initial pose inaccuracies and noisy sensor data.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on knowing their precise location (ego-pose) to navigate safely and effectively. While GPS systems offer a starting point, their accuracy can be limited, especially in urban environments. To overcome this, autonomous vehicles often use pre-built 3D maps as an additional sensor. The task of "relocalization" is to accurately determine the vehicle's pose within this map, even when the initial GPS estimate is significantly off (large translational and rotational drifts). Traditional methods like Iterative Closest Point (ICP) struggle with these large initial errors and can be computationally expensive, especially when dealing with dynamic objects (like other cars) or temporal inconsistencies in the map. This paper aims to bridge this gap by proposing a robust and efficient deep learning solution for 3D-to-3D relocalization.

Methods

StickyLocalization employs a novel end-to-end architecture that jointly learns keypoint detection and matching. The core components are:

  • Pillar Layer: This layer processes raw 3D point clouds. It first encodes local geometric features around potential keypoints into "pillars" (cylinder-shaped regions). A positional encoder then captures global context. Crucially, a "Pillar Filtering" step adaptively selects the most distinctive keypoints, moving away from pre-defined feature extractors.
  • Graph Neural Network (GNN) Layer: The selected keypoints and their features are fed into a GNN. This GNN constructs a multiplex graph, allowing for extensive self- and cross-attention mechanisms. This enables the network to aggregate both local and global contextual information, learning relationships between points within a cloud and between the current scan and the map.
  • Optimal Transport Layer: This layer refines the correspondences identified by the GNN. It uses a soft-assignment mechanism to produce a probability distribution over potential matches.
  • Novel Distance-Based Matching Loss: A key innovation is the introduction of a new loss function. Unlike traditional hard matching losses that penalize slightly off-but-correct matches, this distance-based loss rewards closer points and penalizes farther ones more gradually. This is crucial for the joint learning of keypoint selection and matching, leading to smoother training and faster convergence.

Datasets

The paper evaluates StickyLocalization on two prominent datasets:

  • nuScenes: Used for relocalization tasks. This dataset features diverse and challenging urban driving scenarios captured in Boston and Singapore, providing ground truth poses from an RTK system. The authors specifically use the Boston-Seaport test set, which is from a completely different environment than the training set (Singapore-Onenorth), to test generalization.
  • KITTI Odometry: Used for loop closure estimation. This dataset provides sequences with overlapping trajectories, allowing for the evaluation of the method's ability to recognize previously visited locations.

Key Insights and Ideas

  • End-to-End Joint Learning: The paper demonstrates the power of jointly learning keypoint detection and matching. This allows the network to learn features that are not only descriptive but also well-suited for the subsequent matching and pose estimation tasks, outperforming methods that rely on pre-selected features.
  • GNNs for Context: The use of GNNs with self- and cross-attention is critical for aggregating rich contextual information from point clouds, enabling robustness against noise and dynamic objects.
  • Distance-Based Loss for Robustness: The novel distance-based matching loss is a significant contribution, directly addressing the challenges of training end-to-end systems that involve feature selection, leading to improved convergence and accuracy.
  • Pillar Representation: The choice of "pillars" as the basic unit for feature extraction and aggregation proves effective for capturing local geometry in a computationally efficient manner.

Key Results

StickyLocalization achieves state-of-the-art performance on both relocalization and loop closure tasks:

  • Relocalization (nuScenes): The method significantly outperforms traditional methods (ICP, G-ICP, FPFH) and other deep learning approaches (D3Feat, 3DFeat-Net) in terms of both translational and rotational errors, especially under challenging "Hard" conditions with large rotational offsets (up to 180 degrees).
  • Loop Closure (KITTI): StickyLocalization demonstrates competitive results, particularly excelling in rotational error reduction compared to other methods.
  • Runtime: A major advantage is its speed. By avoiding computationally expensive post-processing steps like RANSAC (common in other deep learning methods) and instead using SVD for pose estimation, StickyLocalization achieves a substantial runtime benefit, making it suitable for real-time applications. For instance, it is significantly faster than D3Feat and 3DFeat-Net on the nuScenes dataset.

Limitations and Future Work

  • Generalization to Unseen Environments: While the paper tests generalization to a different environment for relocalization (Singapore training, Boston testing), the extent of its generalization to vastly different environments or sensor types is not fully explored.
  • Dynamic Object Handling: While the method is more robust to dynamic objects than traditional ICP, explicitly handling and removing dynamic elements could further improve performance.
  • Future Work: The authors suggest that the expressive descriptors learned by the GNN could be utilized for other perception tasks, hinting at broader applications beyond relocalization. Further exploration of different GNN architectures and attention mechanisms could also yield improvements.

Summary updated

BibTeX
@inproceedings{fischer2022stickylocalization,
  author = {Kai Fischer and Martin Simon and Stefan Milz and Patrick Mäder},
  title = {{StickyLocalization: Robust End-to-End Relocalization on Point Clouds Using Graph Neural Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure