Abstract
Visual place recognition is one of the essential and challenging problems in the fields of robotics. In this letter, we for the first time explore the use of multi-modal fusion of semantic and visual modalities in dynamics-invariant space to improve place recognition in dynamic environments. We achieve this by first designing a novel deep learning architecture to generate the static semantic segmentation and recover the static image directly from the corresponding dynamic image. We then innovatively leverage the spatial-pyramid-matching model to encode the static semantic segmentation into feature vectors. In parallel, the static image is encoded using the popular Bag-of-words model. On the basis of the above multi-modal features, we finally measure the similarity between the query image and target landmark by the joint similarity of their semantic and visual codes. Extensive experiments demonstrate the effectiveness and robustness of the proposed approach for place recognition in dynamic environments.
Introduction
Visual Place Recognition (VPR) serves as the fundamental "Have I been here before?" mechanism for autonomous robotic navigation. While existing VPR systems perform reliably in static environments, dynamic real-world scenarios—cluttered with moving vehicles and pedestrians—present significant challenges. Traditional approaches typically rely on masking dynamic objects or using Generative Adversarial Networks (GANs) for image translation. However, masking often discards critical spatial information, while GAN-based "cleaning" can introduce blurry artifacts that degrade feature extraction.
This research proposes a "dynamics-invariant perception space" that simultaneously recovers the visual appearance and semantic structure of the underlying static scene. By "hallucinating" a static version of a dynamic scene, the system ensures robust recognition even when large dynamic objects dominate the field of view.
The MMDIP Framework
The proposed Multi-modal Dynamics-Invariant Perception (MMDIP) network utilizes a multi-stage pipeline to bridge the gap between dynamic observations and static landmarks.
The architecture consists of three core components:
- Static Semantic Reconstruction: A modified ERFNet generates a semantic probability map. Unlike standard segmentation, this module predicts static labels (e.g., road, building) for pixels currently obscured by dynamic objects.
- Static Image Recovery (SSGGNet): A conditional GAN (cGAN) takes the dynamic image and the semantic probability map as inputs. The semantic map serves as a structural prior, guiding the generator to fill occluded regions with contextually appropriate textures.
- Multi-modal Feature Encoding: The system extracts features from both modalities. The recovered static image is encoded via a Bag-of-Words (BoW) model, while the semantic map is encoded using Spatial Pyramid Matching (SPM). SPM is crucial as it preserves the global spatial layout of the scene, distinguishing between similar-looking environments.
Experimental Evaluation
The framework was evaluated using the CARLA simulator, which provided 12,691 pairs of static-dynamic images with perfect ground truth for occluded regions. The dataset was split into 10,000 training pairs and 2,691 testing pairs.
Quantitative results demonstrate that the MMDIP approach significantly outperforms existing baselines. The proposed method achieved a Recall@1 of 80.86%, compared to 65.91% for Pix2Pix-BoW and 70.68% for SRMGAN-BoW. Ablation studies highlighted the importance of spatial structure; increasing the pyramid level from $L=0$ to $L=2$ in the SPM module improved Recall@1 from 72.17% to 80.86%. Furthermore, the pipeline maintains near real-time performance, operating at approximately 18 frames per second (fps) on an NVIDIA TITAN Xp.
Conclusion
This work introduces a novel multi-modal approach to VPR by operating in a dynamics-invariant perception space. By combining semantic consistency with visual recovery, the MMDIP framework achieves high robustness in dynamic environments. Future research will focus on addressing the sim-to-real gap, handling long-term environmental changes such as seasonal variations, and integrating the system into full Semantic SLAM pipelines.
Summary updated