Benchmarking Visual Localization for Autonomous Navigation

Lauri Suomela, Atakan Dag, Harry Edelman, Joni‐Kristian Kämäräinen, Kämäräinen, Joni-Kristian

TL;DR

This paper introduces a novel simulator-based benchmark for evaluating visual localization methods within the context of autonomous navigation. By integrating state-of-the-art visual localization techniques into a complete navigation stack and proposing a new "failure rate" metric, the authors reveal critical insights into how well these methods perform under realistic, dynamic conditions, highlighting significant differences in their practical utility.

What Problem Does This Paper Address?

Autonomous navigation, particularly for robots and vehicles, heavily relies on knowing "where am I?" This is where visual localization comes in, using camera images to estimate a robot's pose (position and orientation) relative to a pre-existing map. While numerous visual localization methods have been developed and benchmarked, these evaluations typically use static datasets and focus solely on accuracy metrics like recall rate. This leaves a critical gap: how do these methods actually perform when integrated into a real-time, closed-loop navigation system, and how do dynamic environmental factors like changing illumination, weather, and viewpoint affect their performance in enabling successful navigation? The authors argue that traditional benchmarks don't adequately answer these questions, potentially leading to methods that perform well on paper but fail in practice.

Methods

The core of this work is a simulator-based benchmark built using the CARLA autonomous driving simulator and a ROS2 port of the Hloc visual localization toolbox. This setup allows for controlled experiments with various environmental factors.

  • ROS2 Visual Localization Package: This package acts as a ROS2 wrapper for the Hloc toolbox, enabling real-time processing of camera streams for pose estimation. It supports various state-of-the-art hierarchical localization methods, combining global place recognition (NetVLAD, Ap-GeM) with local feature extractors (SIFT, D2-net, R2D2, SuperPoint).
  • Navigation Stack Integration: The visual localization output is fed into a motion planning and control system. This includes a global planner (from CARLA), a custom local planner, and a controller composed of PID controllers for steering.
  • Sensor Fusion: To achieve the high-frequency pose updates required by the controller, visual localization estimates are fused with simulated wheel odometry using an Extended Kalman Filter (EKF). This also provides a fallback when visual localization fails.
  • Novel Metric: Failure Rate: Beyond traditional recall rate, the authors introduce a "failure rate" metric. This measures the number of re-initializations required to complete a route over multiple episodes, directly reflecting how often the navigation system fails due to localization errors. This metric is crucial for understanding practical navigation success.

Datasets

The benchmark utilizes the CARLA simulator, specifically two default maps: Town01 (a small town) and Town10 (a larger city). For each map, a 1.2 km (Town01) and 0.5 km (Town10) test route was defined. Gallery sets of images were captured by driving these routes with CARLA's autopilot, and these gallery sets were used to build 3D reconstructions. The experiments then involved generating "query" images under various simulated conditions (illumination, viewpoint, weather) to test the localization methods.

Key Insights and Ideas

  • Real-world relevance of benchmarks: Traditional static datasets may not accurately reflect the performance of visual localization methods in dynamic, closed-loop navigation scenarios.
  • The importance of a failure rate metric: Recall rate alone is insufficient. A method might have high recall but still cause catastrophic navigation failures due to infrequent but severe localization errors. The "failure rate" metric directly addresses this.
  • SuperPoint's remarkable robustness: The SuperPoint local feature extractor, when paired with NetVLAD or Ap-GeM, consistently outperformed other methods across all tested conditions, especially under significant illumination and viewpoint changes.
  • Navigation performance threshold: There's a critical threshold for visual localization accuracy (recall rate) below which improvements offer little to no benefit for navigation. For instance, in Town01, recall at threshold T1 needs to be above 60% to significantly improve navigation over wheel odometry alone.
  • Weather is a significant challenge: Weather conditions, particularly fog, proved to be extremely challenging for most methods, with only SuperPoint showing some resilience.

Key Results

  • Illumination: As illumination conditions worsened (approaching darkness), failure rates increased for all methods. SuperPoint consistently achieved the lowest failure rates, even in near-darkness.
  • Viewpoint: Similar to illumination, viewpoint changes increased failure rates. SuperPoint again showed superior robustness, maintaining performance even with significant vertical offsets and pitch changes.
  • Weather: Fog significantly degraded performance for most methods. SuperPoint was the only method that showed a consistent improvement in navigation performance across varying visibility conditions.
  • Method Comparison: SuperPoint, paired with NetVLAD or Ap-GeM, was the top-performing combination. SIFT, despite its age, remained competitive. D2-net performed the worst.
  • Correlation: A strong correlation was observed between the failure rate and recall rate, reinforcing the idea that sufficient localization accuracy is a prerequisite for successful navigation.

Limitations and Future Work

  • Synthetic Data: The benchmark relies on synthetic data from CARLA. While this allows for controlled experiments and ground truth, it may not perfectly replicate the complexities and nuances of real-world sensor data and environments. The authors acknowledge that synthetic data might be "easier" for localization methods.
  • Limited Scope of Experiments: While the benchmark supports various factors (Table 1), the paper primarily focuses on illumination, viewpoint, and weather changes.
  • Future Work: The authors suggest extending the benchmark to investigate sequential visual localization (using image sequences rather than single images), camera placement effects, and dynamic objects. They also propose testing other localization method families beyond hierarchical approaches.

Summary updated

Abstract

This work introduces a simulator-based benchmark for visual localization in the autonomous navigation context. The dynamic benchmark enables investigation of how variables such as the time of day, weather, and camera perspective affect the navigation performance of autonomous agents that utilize visual localization for closed-loop control. The experimental part of the paper studies the effects of four such variables by evaluating state-of-the-art visual localization methods as part of the motion planning module of an autonomous navigation stack. The results show major variation in the suitability of the different methods for vision-based navigation. To the authors' best knowledge, the proposed benchmark is the first to study modern visual localization methods as part of a complete navigation stack. We make the benchmark available at https://github.com/lasuomela/carla_vloc_benchmark.
BibTeX
@inproceedings{suomela2022benchmarking,
  author = {Lauri Suomela and Atakan Dag and Harry Edelman and Joni‐Kristian Kämäräinen and Kämäräinen, Joni-Kristian},
  title = {{Benchmarking Visual Localization for Autonomous Navigation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure