A Simulation Benchmark for Vision-based Autonomous Navigation

Lauri Suomela, Atakan Dag, Harry Edelman, Joni-Kristian Kämäräinen

Abstract

Traditional visual localization research relies heavily on static image datasets and recall-based metrics. However, these metrics often fail to capture how localization errors propagate through an autonomous vehicle's navigation stack. This paper introduces a simulator-based benchmark using the CARLA environment to evaluate localization in a closed-loop system. By testing various local features and hierarchical matching strategies, we identify a critical "tipping point" for localization utility and demonstrate that learned features like SuperPoint significantly outperform traditional methods in adverse conditions such as low light and extreme viewpoint changes.

Introduction

Visual localization—the process of determining a robot's precise position and orientation using camera data—is a cornerstone of autonomous navigation. Historically, the field has evaluated performance using "recall" on static datasets: the percentage of query images localized within a specific distance threshold of the ground truth. While useful for benchmarking algorithms in isolation, high recall does not inherently guarantee a functional robot. In real-world scenarios, localization errors feed directly into motion planners and controllers. A single failure during a critical maneuver, like a sharp turn, is far more detrimental than multiple small errors on a straight path. This paper addresses the research gap between static accuracy and operational success by proposing a closed-loop simulation benchmark.

The Simulation Framework

To bridge the gap between image matching and autonomous driving, we developed a benchmark using the CARLA simulator and a custom ROS2-wrapper for the Hloc (Hierarchical Localization) toolbox. The system operates in two distinct phases: mapping and navigation.

Overview of the visual localization and navigation pipeline within the CARLA simulator.

During the Mapping phase, a vehicle traverses a route under ideal conditions to create a reference "gallery" of images and a 3D point cloud via Structure-from-Motion (SfM). In the Navigation phase, the vehicle attempts the same route under challenging environmental conditions. The navigation stack utilizes hierarchical methods for localization, retrieving similar images via NetVLAD or Ap-GeM before performing local feature matching with algorithms such as SuperPoint, SIFT, R2D2, or D2-net. An Extended Kalman Filter (EKF) fuses these visual pose estimates with noisy wheel odometry, allowing the vehicle to maintain a trajectory even during temporary visual failures. Finally, a PID controller manages steering and acceleration based on waypoints from a global planner.

Beyond Static Metrics: The Failure Rate

We propose a more meaningful metric for robotics: the Failure Rate, defined as the number of re-initializations (resets required after a crash or departure from the route) per kilometer. Our experiments in CARLA's Town01 and Town10 environments revealed a clear "tipping point" for localization utility. For instance, in Town01, a localization method must achieve at least 60% recall at strict thresholds to provide a net benefit over simple wheel odometry. Below this threshold, inaccurate visual data can actually degrade performance, leading to higher failure rates than if the vehicle relied solely on odometry.

Robustness Analysis

Our benchmark evaluated several local feature extractors across three primary axes of difficulty:

  • Illumination: As sun intensity decreased, learned features like SuperPoint (paired with SuperGlue) maintained near-zero failure rates. In contrast, traditional features like SIFT and newer methods like D2-net saw failure rates spike as soon as lighting deviated significantly from the sunny gallery set.
  • Viewpoint Changes: We simulated camera pitch and vertical offsets to mimic different vehicle types. SuperPoint demonstrated remarkable resilience, continuing to function at offsets where all other methods reverted to the baseline failure rate of wheel odometry.
  • Weather and Aliasing: Fog acted as a significant performance equalizer, with most methods struggling when visibility dropped below 30 meters. Additionally, in urban environments like Town10, repetitive textures on skyscrapers led to "perceptual aliasing," where the system incorrectly localized the vehicle to a different part of the street, highlighting a unique challenge for city-scale navigation.

Conclusion and Future Work

This benchmark demonstrates that closed-loop evaluation is essential for understanding the true utility of visual localization. While synthetic data in simulators may be cleaner than real-world imagery, the trends observed—particularly the robustness of learned features and the existence of a performance tipping point—provide vital insights for designing safer autonomous systems. Future work will explore sequential localization to leverage temporal data and investigate the impact of multi-camera configurations on navigation stability.

Summary updated

BibTeX
@inproceedings{suomela2022simulation,
  author = {Lauri Suomela and Atakan Dag and Harry Edelman and Joni-Kristian Kämäräinen},
  title = {{A Simulation Benchmark for Vision-based Autonomous Navigation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure