Map-Free Visual Relocalization Enhanced by Instance Knowledge and Depth Knowledge

Mingyu Xiao, Runze Chen, Haiyong Luo, Fang Zhao, Fan Wu, Hao Xiong, Xuepeng Ma, Juan WangICASSP 2025 · DOI

TL;DR

This paper introduces a novel map-free visual relocalization method that significantly improves camera pose estimation accuracy by integrating instance-level semantic knowledge and depth information. By focusing feature matching on specific objects and using depth to recover scale, the approach achieves state-of-the-art performance, outperforming existing methods by a substantial margin.

What Problem Does This Paper Address?

Visual relocalization, the task of determining a camera's position and orientation within an environment, is crucial for applications like augmented reality and robotics. Traditional map-based methods require extensive 3D maps and image datasets, limiting their scalability. Map-free visual relocalization aims to overcome this by estimating pose using only a query image and a reference image, eliminating the need for pre-built maps. However, this approach faces significant challenges: accurately matching feature points between images and recovering the correct scale from monocular images, which can lead to substantial rotational and metric errors, ultimately causing localization failures.

Methods

The proposed method enhances map-free visual relocalization through two key innovations:

  • Instance-Enhanced Matching: The authors leverage an existing multi-view stereo reconstruction model (DUSt3R) for initial feature point matching. To address DUSt3R's suboptimal performance in map-free relocalization, they introduce instance segmentation (using SegGPT) to guide the matching process. Instead of matching features globally, the method focuses on matching within identified object instances. This "instance-level semantic matching" restricts the search space, reducing incorrect matches between different objects and improving the robustness of feature point correspondences. For background areas, a global matching strategy is still employed.
  • Depth-Enhanced Scale Recovery: To address the scale ambiguity inherent in monocular images, the method incorporates depth estimation. They utilize the Metric3D model to predict depth information from a single image. This depth knowledge is then used to project feature points into 3D space, allowing for the recovery of a scaled translation vector. By combining the estimated depth with matching point pairs, they compute a scale factor for the translation vector, significantly improving metric accuracy.

Datasets

The paper evaluates its method on the map-free visual relocalization dataset introduced by Arnold et al. [16].

Key Insights and Ideas

  • Instance-level matching is crucial for map-free relocalization: While global feature matching is useful, focusing on matching within specific object instances significantly reduces errors, especially in scenarios with parallax and temporal differences. This is because foreground objects are more robust features for global scene understanding.
  • Depth estimation is key to recovering scale: The inherent scale ambiguity in monocular images is a major hurdle. By integrating depth estimation, the method can accurately recover the scale of the translation vector, leading to more precise metric localization.
  • Combining instance and depth knowledge synergistically: The paper demonstrates that both instance knowledge and depth knowledge are essential and work together to achieve superior performance. Removing either component leads to a noticeable degradation in accuracy.

Key Results

The proposed method achieves state-of-the-art results on the map-free visual relocalization dataset. Specifically:

  • Average Median Translation Error: Reduced to 0.596m, a significant improvement over the previous best method (RPR[3D-3D] at 1.667m).
  • Average Median Rotation Error: Reduced to 9.030°, also a substantial improvement over RPR[3D-3D] at 22.623°.
  • The method also shows superior performance in Average Median Precision Reprojection Error and AUC@VCRE < 90px.
  • The paper also highlights that their method was a winner in the Map-free Workshop & Challenge (ECCV2024).

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method significantly improves accuracy, the performance of DUSt3R's initial matching can still be a bottleneck. The visualization in Figure 1 shows that the confidence in foreground object matching, while improved by instance knowledge, is still not "significant."
  • Future Work: Not explicitly detailed in the provided text, but the success of combining instance and depth knowledge suggests further exploration into integrating other semantic or geometric cues could yield even better results. The authors also mention their method's superiority in generalization, implying its potential for diverse and challenging real-world scenarios.

Summary updated

BibTeX
@inproceedings{xiao2025map,
  author = {Mingyu Xiao and Runze Chen and Haiyong Luo and Fang Zhao and Fan Wu and Hao Xiong and Xuepeng Ma and Juan Wang},
  title = {{Map-Free Visual Relocalization Enhanced by Instance Knowledge and Depth Knowledge}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888917},
}
Figure