Introduction
In Visual Simultaneous Localization and Mapping (V-SLAM), Loop Closure Detection (LCD) is essential for correcting the cumulative drift inherent in odometry. By recognizing a previously visited location, a robot can re-align its trajectory and maintain a consistent global map. However, the cost of a "false positive"—incorrectly identifying a new location as a visited one—is high, often leading to total map corruption.
To mitigate this risk, Geometric Verification (GV) is employed as a secondary filter. After a global retrieval stage identifies potential matches, GV utilizes local feature matching to confirm spatial consistency, typically through RANSAC-based estimation. While local features have been benchmarked extensively for tasks like homography estimation, their performance as a binary classifier in long-term scenarios (varying weather, seasons, and lighting) has lacked a unified evaluation framework. This paper presents GV-Bench, a modular benchmark designed to fill this gap.
The GV-Bench Framework
GV-Bench implements a standardized LCD pipeline to evaluate local feature matchers in a realistic context. The pipeline consists of two primary stages:
- Retrieval Stage: A global descriptor (NetVLAD) is used to retrieve the top $N$ candidate images from a database.
- Verification Stage: The candidate pairs are processed by a local feature matching algorithm to determine if they represent a true loop closure.
Evaluation Metrics
The benchmark prioritizes the Max Recall @ 100% Precision (MR) metric. In robotics, achieving zero false positives is often more critical than finding every possible loop. MR measures the maximum percentage of true loops a system can identify before a single false positive is accepted. Additionally, the framework tracks Average Precision (AP) to evaluate the overall ranking capability of the matchers.
Experimental Setup
The researchers evaluated six representative local feature matching combinations:
- Handcrafted: SIFT with Nearest Neighbor (NN) matching.
- Sparse Learning-based: SuperPoint+NN, SuperPoint+SuperGlue (SP+SG), DISK+NN, and DISK+LightGlue (DISK+LG).
- Detector-free: LoFTR, which performs dense matching without explicit keypoint detection.
These methods were tested across six sequences derived from three major datasets: Oxford RobotCar (weather and seasonal changes), Nordland (extreme summer-to-winter transitions), and UAcampus (extreme day-to-night illumination and viewpoint shifts).
Results and Key Insights
The benchmarking results reveal several critical insights into the current state of local feature matching for GV:
Performance Leaders
SuperPoint + SuperGlue (SP+SG) emerged as the most reliable method for high-precision robotics applications, achieving the highest Max Recall in four of the six sequences and an average MR of 44.4%. While LoFTR demonstrated the highest Average Precision (93.2%), indicating superior ranking ability, it struggled to maintain 100% precision at high recall levels compared to SP+SG. Modern learning-based methods significantly outperformed the classic SIFT+NN, which struggled particularly in the Nordland and Night sequences.
The Challenge of Extreme Conditions
Day-to-night transitions remain a significant hurdle for all tested methods. In the "Night" sequences, many true loop closures resulted in zero successful matches. The radical appearance changes caused by artificial lighting and deep shadows continue to baffle even modern learning-based descriptors.
Perceptual Aliasing and Ground Truth Ambiguity
In repetitive environments like the Nordland railway, "perceptual aliasing" often leads to RANSAC failures. False matches can appear spatially consistent by chance, tricking the verification system. Furthermore, the authors noted that standard GPS-based ground truth can be misleading; some pairs labeled as "false" due to distance thresholds were visually identical, suggesting that current SLAM benchmarks may need more nuanced labeling strategies.
Conclusion
GV-Bench provides a rigorous framework for evaluating local features in the context of long-term robot autonomy. While learning-based methods like SuperPoint+SuperGlue represent a significant leap over handcrafted methods like SIFT, achieving robust, zero-false-positive verification in all environmental conditions remains an open challenge. Future work should focus on training descriptors with more diverse "long-term" data and developing advanced outlier rejection techniques beyond standard RANSAC.
The GV-Bench code and datasets are available at: https://github.com/jarvisyjw/GV-Bench
Summary updated