TL;DR
This paper introduces AmsterTime, a new benchmark dataset for visual place recognition (VPR) that specifically addresses the challenge of severe domain shift between historical archival images and modern street-view imagery. By evaluating various VPR methods on this dataset, they highlight the effectiveness of models pre-trained on large landmark datasets for achieving high performance, even with self-supervised learning approaches.
What Problem Does This Paper Address?
Visual Place Recognition (VPR) is the task of identifying the geographical location of an image, crucial for applications like robotics, autonomous navigation, and historical image archiving. A key challenge in VPR is "domain shift" – the significant differences in appearance, viewpoint, camera, and lighting conditions between images of the same place captured at different times or with different technologies. Existing VPR datasets often struggle to capture this severe domain shift, particularly when comparing historical imagery with contemporary street-view data. This paper aims to fill this gap by creating a challenging benchmark dataset, AmsterTime, specifically designed to test VPR models under these extreme conditions.
Methods
The core of this paper is the introduction of the AmsterTime dataset. The authors employed a novel crowdsourcing approach using a custom web application that integrated historical archival images of Amsterdam with a 3D street-view navigator from the Mapillary platform. Human annotators verified image pairs, ensuring accurate matches of the same locations across different eras.
Beyond dataset creation, the paper evaluates a range of VPR methods:
- Local Image Features: Traditional methods like SIFT and LIFT, aggregated using Bag-of-Visual-Words (BoVW).
- Off-the-shelf CNN Models: Pre-trained image classification models (VGG-16, ResNet-50, ResNet-101) used as feature extractors.
- VPR-specific Models: Pre-trained NetVLAD and AP-GeM models, known for their effectiveness in place recognition.
- Self-supervised Learning: Various SimSiam models trained with different backbones and datasets, demonstrating the potential of learning representations without explicit labels.
- Supervised Learning: A ResNet-18 model trained with triplet loss on ground-truth pairs to establish a supervised baseline and quantify the "supervision gap."
To understand what the models are learning, they utilize Grad-CAM for visual explanations, highlighting the regions in images that contribute most to a model's decision.
Datasets
- AmsterTime (New Benchmark): A collection of approximately 2,500 image pairs, featuring historical archival images of Amsterdam matched with contemporary street-view images from Mapillary. This dataset is designed to represent a severe domain shift.
- ImageNet: Used for pre-training standard CNN architectures.
- Google Landmarks Dataset (GLDv2): A large-scale dataset used for pre-training VPR-specific models and for creating a subset for visualization experiments.
- Piccadilly Dataset: Used for training LIFT descriptors.
- Pittsburgh250k: Used for pre-training NetVLAD.
Key Insights and Ideas
- Severe Domain Shift is Crucial: The AmsterTime dataset effectively demonstrates that VPR models struggle significantly when faced with drastic changes in appearance, viewpoint, and time, a scenario common in real-world applications involving historical data.
- Pre-training on Landmarks is Key: Models pre-trained on large, diverse landmark datasets (like Google Landmarks) show superior performance on AmsterTime, even outperforming models trained on AmsterTime itself with self-supervision. This suggests that broad visual knowledge of landmarks is highly beneficial.
- Self-Supervision Shows Promise: Despite the challenges, self-supervised methods like SimSiam can learn meaningful representations from AmsterTime, achieving competitive results, especially when using robust backbones. This is important given the scarcity of labeled data for such domain shifts.
- Visual Explanations Reveal Learning: Grad-CAM visualizations confirm that models, particularly those trained on landmarks, learn to focus on salient architectural features and landmarks, indicating they are indeed learning relevant visual cues for place recognition.
Key Results
- Best Performance: The ResNet-101 model pre-trained on the Google Landmarks dataset achieved the highest accuracy, reaching 84% for verification and 24% for retrieval (Top-1 accuracy).
- Supervised vs. Self-supervised: A supervised ResNet-18 model trained on AmsterTime showed a gap of only around 7% in mAP compared to unsupervised models, indicating the effectiveness of self-supervised learning on this challenging dataset.
- VGG-16 vs. ResNet: Interestingly, VGG-16 pre-trained on ImageNet performed better than ResNet-50 and ResNet-101 on verification tasks, and significantly better on retrieval tasks, suggesting architectural differences can play a role.
- Local Features: Traditional local features (SIFT, LIFT) with BoVW performed poorly, achieving only around 57-58% accuracy on verification, highlighting the need for more robust deep learning features.
Limitations and Future Work
- Dataset Size: While curated, AmsterTime has a limited number of pairs (~2,500) compared to some other VPR datasets. The authors acknowledge this and use it to motivate self-supervised learning.
- Annotation Bias: Although crowdsourced and verified, there might be inherent biases in the selection of archival images or the annotators' familiarity with certain areas.
- Future Work: The authors suggest further exploration of self-supervised methods, investigating more advanced domain adaptation techniques, and potentially expanding the dataset with more diverse historical periods and locations. The visual explanations also open avenues for understanding model failures and biases.
Summary updated