Abstract
Visual Place Recognition (VPR) in areas with similar scenes such as urban or indoor scenarios is a major challenge. Existing VPR methods using global descriptors have difficulty capturing local specific regions (LSR) in the scene and are therefore prone to localization confusion in such scenarios. As a result, finding the LSR that are critical for location recognition becomes key. To address this challenge, we introduced Patch-NetVLAD+, which was inspired by patch-based VPR researches. Our method proposed a fine-tuning strategy with triplet loss to make NetVLAD suitable for extracting patch-level descriptors. Moreover, unlike existing methods that treat all patches in an image equally, our method extracts patches of LSR, which present less frequently throughout the dataset, and makes them play an important role in VPR by assigning proper weights to them. Experiments on Pittsburgh30k and Tokyo247 datasets show that our approach achieved up to 6.35% performance improvement than existing patch-based methods.
Introduction
Visual Place Recognition (VPR) is the fundamental "where am I?" problem for computer vision. While global descriptors—which summarize an entire image into a single vector—work well for general scenes, they often fail in "perceptually aliasing" environments. These are places that look very similar, such as different street corners in a city or generic office hallways.
The core issue is that global descriptors struggle to distinguish between Local Specific Regions (LSR)—unique landmarks like a specific sign—and common, non-informative regions like the sky or pavement. Existing patch-based methods help by looking at local details, but they often treat every patch as equally important and use descriptors that were not specifically trained to represent small image patches.
Methodology
Patch-NetVLAD+ introduces two primary improvements over the existing Patch-NetVLAD framework to enhance recognition accuracy.
Fine-tuned Patch Descriptors
Instead of using a NetVLAD model trained on full images, we fine-tune the model using a triplet loss specifically for patches. To identify corresponding training data (since datasets usually only provide image-level GPS labels), we utilize a combination of patch-to-patch similarity and keypoint matching (GCRANSAC). This ensures that the model learns to "pull" corresponding patches together in the feature space while "pushing" non-matching ones apart.
Weighted Matching Strategy
This is the paper's most novel contribution. We argue that patches appearing frequently across a dataset (like the sky or road) are less useful for recognition. As shown in the conceptual diagram below, we use K-means to cluster patch descriptors from the entire database. Patches that are far from these cluster centroids are considered "special" or rare (LSR).
During the matching phase, these rare patches are assigned higher weights, ensuring they play a bigger role in the final similarity score than generic features. This hierarchical approach first finds the top 100 candidates using global descriptors before applying heavy-duty weighted patch matching to refine the results.
Experimental Evaluation
We evaluated Patch-NetVLAD+ on two major VPR benchmarks: Pittsburgh30k (urban viewpoint changes) and Tokyo247 (significant illumination and structural similarity).
Key Results
- Accuracy Boost: On Pittsburgh30k, the method achieved 88.50% Recall@1, a 1.1% improvement over the original Patch-NetVLAD.
- Dominance in Tough Scenes: The improvement was most dramatic on the Tokyo247 dataset, where Recall@1 jumped to 83.49%, compared to 77.14% for the previous state-of-the-art—a significant 6.35% lead.
- Robustness: Qualitative results show that while older methods get "distracted" by matching patches of sky, Patch-NetVLAD+ consistently finds matches on buildings, signs, and unique architectural features.
Conclusion
Patch-NetVLAD+ demonstrates that prioritizing unique local landmarks over common features significantly improves VPR performance. While the weighted matching strategy is effective, future work could incorporate semantic contextual information—explicitly teaching the model that "signage" is more important than "vegetation"—to further refine the weighting process and improve computational efficiency.
Summary updated