Attention-Enhanced Cross-modal Localization Between 360 Images and Point Clouds

Zhipeng Zhao, Huai Yu, Chenwei Lyv, Wen Yang, Sebastian Scherer

TL;DR

This paper introduces an end-to-end learnable framework for localizing 360-degree panoramic images within 3D LiDAR maps. By combining spherical-oriented convolutions with an attention mechanism, the model bridges the "heterogeneity gap" between 2D textures and 3D geometry, significantly outperforming traditional perspective-camera approaches in complex urban environments.

What Problem Does This Paper Address?

Visual localization—determining where a robot or vehicle is based on what it sees—is a critical backup for when GNSS (GPS) fails. While matching images to existing LiDAR maps is cost-effective, it faces two major hurdles:

  1. The Field-of-View (FOV) Mismatch: Standard pinhole cameras see only a narrow slice of the world, whereas LiDAR sensors capture a full 360-degree sweep. This makes it difficult to find enough overlapping features to ensure a confident match.
  2. The Heterogeneity Gap: Images record color and texture, while point clouds record spatial coordinates and geometry. Comparing these two "languages" in a high-dimensional feature space is inherently difficult.

Furthermore, when you project a 360-degree view onto a flat 2D image (an equirectangular projection), it becomes heavily distorted—objects near the "poles" look stretched compared to those at the "equator." Standard Convolutional Neural Networks (CNNs) struggle to recognize these distorted features.

Methods

The authors propose a dual-branch "Attention-Enhanced" network to extract and align features from both modalities:

  • Spherical-Oriented CNN: To handle the distortion of 360 images, the image branch uses a modified ResNet-18. Instead of standard fixed-grid convolutions, it uses spherical convolutions that adjust their sampling locations based on the image's latitude, effectively "unwarping" the distortion during feature extraction.
  • Point Cloud Branch: This branch utilizes PointNet to process raw 3D coordinates from LiDAR sub-maps.
  • Attention Enhancement: Both branches incorporate Squeeze-and-Excitation (SE) blocks. This mechanism allows the network to "decide" which feature channels are most important for cross-modal matching (e.g., focusing on building facades rather than the sky or road surface) by adaptively re-weighting them.
  • Global Aggregation: The local features from both branches are fed into a NetVLAD layer, which aggregates them into a single, compact global descriptor.
  • Metric Learning: The model is trained using a combination of triplet losses and a specific "anchor loss" to ensure that an image and its corresponding point cloud are pulled closer together in the embedding space than non-matching pairs.

Datasets

The researchers utilized the KITTI-360 dataset, a large-scale outdoor driving dataset.

  • Preprocessing: They stitched dual-fisheye camera images into 360-degree equirectangular panoramas and accumulated LiDAR scans into 3D sub-maps.
  • Augmentation: To ensure robustness, they applied various augmentations to simulate different lighting and weather conditions.
  • Benchmark: They established a new evaluation protocol on KITTI-360, selecting specific sequences for training and testing to ensure no overlap between the two.

Key Insights and Ideas

  • FOV Symmetry is Key: The most significant "aha" moment is that 360-degree images are fundamentally better suited for LiDAR-based localization than perspective images. Because their FOV matches the LiDAR's omni-directional nature, the model can leverage landmarks in every direction simultaneously.
  • Salience Matters: Not all parts of a scene are equally useful for localization. The attention mechanism helps the model ignore "geometric degeneracy" (like flat roads or repetitive trees) and focus on unique structural features that appear in both the image and the point cloud.
  • Distortion Invariance: By modifying the convolution kernels to be "sphere-aware," the model maintains high performance regardless of where an object appears in the panoramic frame.

Key Results

  • Accuracy Boost: The proposed AE-Spherical Model achieved a Recall@1 of 46.23%, a significant improvement over the ResNet-based baseline (36.79%).
  • 360 vs. Perspective: In head-to-head tests, 360 images significantly outperformed perspective images for cross-modal matching, proving that the extra visual information is worth the computational cost of processing panoramas.
  • Ablation Success: Removing either the spherical convolution or the attention mechanism led to a drop in performance, confirming that both components are necessary to bridge the cross-modal gap.

Limitations and Future Work

The authors note that the model performs less effectively in highway scenarios compared to downtown areas. Highways often lack the distinct semantic and geometric information (like unique buildings) found in urban centers, making it harder to find "salient" features.

Future work aims to further strengthen the coupling between the two branches, perhaps by incorporating more explicit semantic information (like object labels) to help the model understand what it is looking at, rather than just matching raw geometric patterns.

Summary updated

Abstract

Visual localization plays an important role for intelligent robots and autonomous driving, especially when the accuracy of GNSS is unreliable. Recently, camera localization in LiDAR maps has attracted more and more attention for its low cost and potential robustness to illumination and weather changes. However, the commonly used pinhole camera has a narrow Field-of-View, thus leading to limited information compared with the omni-directional LiDAR data. To overcome this limitation, we focus on correlating the information of 360 equirectangular images to point clouds, proposing an end-to-end learnable network to conduct cross-modal visual localization by establishing similarity in high-dimensional feature space. Inspired by the attention mechanism, we optimize the network to capture the salient feature for comparing images and point clouds. We construct several sequences containing 360 equirectangular images and corresponding point clouds based on the KITTI-360 dataset and conduct extensive experiments. The results demonstrate the effectiveness of our approach.
BibTeX
@misc{zhao2022attention,
  author = {Zhipeng Zhao and Huai Yu and Chenwei Lyv and Wen Yang and Sebastian Scherer},
  title = {{Attention-Enhanced Cross-modal Localization Between 360 Images and Point Clouds}},
  year = {2022},
  eprint = {2212.02757},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.02757},
}
Figure