DASGIL: Domain Adaptation for Semantic and Geometric-aware Image-based Localization

Hanjiang Hu, Ming Cheng, Zhe Liu, Hesheng Wang

Abstract

Long-Term visual localization under changing environments is a challenging problem in autonomous driving and mobile robotics due to season, illumination variance, etc. Image retrieval for localization is an efficient and effective solution to the problem. In this paper, we propose a novel multi-task architecture to fuse the geometric and semantic information into the multi-scale latent embedding representation for visual place recognition. To use the high-quality ground truths without any human effort, the effective multi-scale feature discriminator is proposed for adversarial training to achieve the domain adaptation from synthetic virtual KITTI dataset to real-world KITTI dataset. The proposed approach is validated on the Extended CMU-Seasons dataset and Oxford RobotCar dataset through a series of crucial comparison experiments, where our performance outperforms state-of-the-art baselines for retrieval-based localization and large-scale place recognition under the challenging environment.

Introduction

Visual localization—the ability of a robot or self-driving car to determine its position based on an image—is a cornerstone of autonomous navigation. However, "long-term" localization is notoriously difficult because environmental factors like seasons, lighting, and weather drastically alter the appearance of a scene. Traditional methods often rely on local features (like SIFT) that fail when pixels change significantly. While deep learning has improved robustness, the most effective models require auxiliary data like depth maps or semantic segmentation. Generating these labels for real-world images is labor-intensive. DASGIL (Domain Adaptation for Semantic and Geometric-aware Image-based Localization) addresses this by training on synthetic data with "perfect" labels and using domain adaptation to bridge the "reality gap."

Methodology

The DASGIL framework utilizes a multi-task architecture designed to bake geometric and semantic "common sense" into its internal features.

Architecture Overview

The core of DASGIL is a one-encoder-two-decoder architecture. A shared "Fusion Feature Extractor" ($E$) feeds into two separate decoders: one for monocular depth prediction ($G_D$) and one for semantic segmentation ($G_S$). This multi-task approach ensures that the latent embeddings capture structural and categorical information that is more stable across environmental changes than raw pixel values.

Overview of the DASGIL framework including training with multi-task generators and the image retrieval pipeline for testing.

Multi-scale Adversarial Training

To handle the domain gap between synthetic (Virtual KITTI) and real (KITTI) images, the authors introduced two types of discriminators for adversarial training:

  • Flatten Discriminator (FD): This discriminator concatenates features from multiple layers to distinguish between real and virtual domains globally.
  • Cascade Discriminator (CD): A more sophisticated module that processes features layer-by-layer. This allows the model to align both low-level textures and high-level concepts across domains, ensuring the extractor produces domain-invariant features.
Architecture of the Flatten Discriminator (FD) and Cascade Discriminator (CD) used for multi-scale adversarial domain adaptation.

Multi-scale Fusion Triplet Loss

For the localization task, which is framed as image retrieval, the model employs a triplet loss applied across multiple feature scales. This ensures that images of the same location are mapped close together in the embedding space, while images of different locations are pushed apart, regardless of environmental variations.

Experimental Evaluation

The model was trained using Virtual KITTI 2 for ground truth labels and the real-world KITTI dataset for domain adaptation. Evaluation was conducted on two challenging benchmarks:

  • Extended CMU-Seasons: Known for drastic changes in foliage and weather.
  • Oxford RobotCar: Includes difficult night and snow scenarios.

Key Results

DASGIL outperformed state-of-the-art baselines like NetVLAD and DenseVLAD:

  • Suburban Areas: On the CMU-Seasons dataset, DASGIL achieved a 12.4% improvement over the next best baseline in coarse precision.
  • Challenging Weather: The model maintained high recall in "Overcast," "Low Sun," and "Snow" conditions on the Oxford RobotCar dataset, where other methods saw significant performance drops.
  • Ablation Studies: The Cascade Discriminator (CD) was found to be particularly effective at aligning feature distributions, leading to superior generalization on real-world test sets.

Conclusion

DASGIL demonstrates that fusing semantic and geometric information creates a robust representation for visual localization. By leveraging domain adaptation, the framework successfully uses "free" labels from simulators to solve real-world problems where manual labeling is prohibitively expensive. While urban environments with many dynamic objects remain a challenge, the multi-scale approach provides a strong foundation for future work in long-term autonomous navigation.

Summary updated

Abstract

Long-Term visual localization under changing environments is a challenging problem in autonomous driving and mobile robotics due to season, illumination variance, etc. Image retrieval for localization is an efficient and effective solution to the problem. In this paper, we propose a novel multi-task architecture to fuse the geometric and semantic information into the multi-scale latent embedding representation for visual place recognition. To use the high-quality ground truths without any human effort, the effective multi-scale feature discriminator is proposed for adversarial training to achieve the domain adaptation from synthetic virtual KITTI dataset to real-world KITTI dataset. The proposed approach is validated on the Extended CMU-Seasons dataset and Oxford RobotCar dataset through a series of crucial comparison experiments, where our performance outperforms state-of-the-art baselines for retrieval-based localization and large-scale place recognition under the challenging environment.
BibTeX
@article{hu2021dasgil,
  author = {Hanjiang Hu and Ming Cheng and Zhe Liu and Hesheng Wang},
  title = {{DASGIL: Domain Adaptation for Semantic and Geometric-aware Image-based Localization}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=pSETrZbpPz9},
}
Figure