Explicit Interaction for Fusion-Based Place Recognition

Jingyi Xu, Junyi Ma, Qi Wu, Zijie Zhou, Yue Wang, Xieyuanli Chen, Ling PeiIROS 2024 · DOI · OpenReview

TL;DR

This paper introduces EINet, a novel fusion-based place recognition network that explicitly leverages the complementary strengths of LiDAR and camera data. By introducing sparse depth supervision from LiDAR to guide camera feature extraction and appearance rendering from cameras to enrich LiDAR features, EINet achieves state-of-the-art performance in recognizing previously visited locations, even across significant time gaps. The authors also release NUSC-PR, a new benchmark for place recognition on the nuScenes dataset, to facilitate future research.

What Problem Does This Paper Address?

Autonomous robots and vehicles often need to recognize their location in GPS-denied environments. Vision-based place recognition (VPR) is cost-effective but susceptible to lighting and weather changes. LiDAR-based place recognition (LPR) is more robust to environmental conditions but lacks rich appearance information. Fusion-based place recognition (FPR) combines both modalities to leverage their respective strengths. However, existing FPR methods often fuse features implicitly, failing to fully exploit the unique advantages each modality offers. This paper addresses the gap in explicitly understanding and utilizing these complementary affordances for more robust and accurate place recognition.

Methods

The core of this work is the EINet (Explicit Interaction Network) architecture. EINet introduces two key explicit interaction mechanisms:

  • Sparse Depth Supervision: LiDAR point clouds are used to provide sparse depth information, guiding the camera branch's encoder-decoder structure to output pseudo-depth maps. This helps the camera features become more geometry-aware, improving robustness over time.
  • Appearance Rendering: Camera images are used to "render" RGB information onto LiDAR point clouds, creating colored point clouds. These are then transformed into range images, enriching the LiDAR features with appearance information, thus improving distinctiveness.

These explicitly interacted features are then fused using a cross-modal transformer. Additionally, a PoseNet is used for auxiliary training to explicitly capture relative poses between query and positive samples, further improving the alignment of features.

Datasets

The paper introduces a new benchmark called NUSC-PR (nuScenes Place Recognition) built upon the publicly available nuScenes dataset. NUSC-PR provides:

  • Standardized data organization for both supervised and self-supervised learning schemes.
  • Evaluation protocols for both paradigms.
  • Data split across multiple cities (Boston-Seaport, Singapore-Onenorth, Singapore-Queenstown, Singapore-Hollandvillage) to test generalization.

Key Insights and Ideas

  • Explicit is Better than Implicit: The paper strongly advocates for explicit interaction between modalities, arguing that it unlocks greater potential than implicit fusion methods that simply combine features.
  • Complementary Strengths: LiDAR's geometric robustness and camera's distinctive appearance are not just combined but actively used to enhance each other. LiDAR's range data helps cameras understand geometry, while camera color data makes LiDAR point clouds more discriminative.
  • Geometry-Aware Features are Key: The sparse depth supervision from LiDAR significantly boosts performance, suggesting that geometric awareness is crucial for long-term place recognition, especially in dynamic environments.
  • NUSC-PR for Standardized Evaluation: The creation of a new benchmark is vital for fair comparison and advancement in the field, especially by including both supervised and self-supervised learning schemes.

Key Results

  • State-of-the-Art Performance: EINet significantly outperforms existing fusion-based methods (AdaFusion, LCPR) and unimodal baselines on the NUSC-PR benchmark. For instance, it achieves a 10.50% improvement in AR@1 over AdaFusion in the supervised learning scheme.
  • Robustness to Time Gaps: EINet demonstrates strong performance in retrieving places with large time gaps (over 120 days).
  • Generalization Ability: The model shows excellent zero-shot generalization to unseen locations, outperforming other fusion-based approaches.
  • Efficient Inference: The total inference time of EINet is around 186.48 ms (5.36Hz), making it suitable for real-time applications.

Limitations and Future Work

  • Computational Cost of Image Decoder: The image decoder, particularly the CVT blocks, is identified as the slowest component, contributing significantly to the overall inference time.
  • Future Work: The authors suggest that the proposed explicit interaction mechanism can be further explored and applied to other multi-modal tasks. The efficient inference also opens doors for deployment in real-world robotic and autonomous driving systems.

Summary updated

Abstract

Fusion-based place recognition is an emerging technique jointly utilizing multi-modal perception data, to recognize previously visited places in GPS-denied scenarios for robots and autonomous vehicles. Recent fusion-based place recognition methods combine multi-modal features in implicit manners. While achieving remarkable results, they do not explicitly consider what the individual modality affords in the fusion system. Therefore, the benefit of multi-modal feature fusion may not be fully explored. In this paper, we propose a novel fusion-based network, dubbed EINet, to achieve explicit interaction of the two modalities. EINet uses LiDAR ranges to supervise more robust vision features for long time spans, and simultaneously uses camera RGB data to improve the discrimination of LiDAR point clouds. In addition, we develop a new benchmark for the place recognition task based on the nuScenes dataset. To establish this benchmark for future research with comprehensive comparisons, we introduce both supervised and self-supervised training schemes alongside evaluation protocols. We conduct extensive experiments on the proposed benchmark, and the experimental results show that our EINet exhibits better recognition performance as well as solid generalization ability compared to the state-of-the-art fusion-based place recognition approaches. Our open-source code and benchmark are released at: https://github.com/BIT-XJY/EINet.
BibTeX
@inproceedings{xu2024explicit,
  author = {Jingyi Xu and Junyi Ma and Qi Wu and Zijie Zhou and Yue Wang and Xieyuanli Chen and Ling Pei},
  title = {{Explicit Interaction for Fusion-Based Place Recognition}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802665},
}
Figure