Abstract
This paper presents the methodology behind the winning entry for the AISG–SLA Visual Localisation Challenge at IJCAI 2023. The core challenge involved reconstructing large-scale urban scenes from sparse image sequences characterized by low overlap and temporal discontinuities. Our approach integrates state-of-the-art deep learning matchers—specifically RoMa and DeDoDe—with the robust Structure-from-Motion (SfM) framework, COLMAP. By employing DINOv2 for global image retrieval and a dense-to-sparse guided matching strategy, we successfully bridged gaps in sequential data, achieving state-of-the-art localization accuracy in complex urban environments.
Introduction
Visual localization—the process of estimating a camera's pose within a known environment—is a fundamental task for autonomous navigation and robotics. However, real-world datasets often present significant challenges that break traditional sequential pipelines. In the AISG-SLA Visual Localisation Challenge, researchers were tasked with processing image sequences from Singapore's urban landscape. These sequences featured low-frequency captures (resulting in minimal overlap between consecutive frames), "time jumps" where data was missing, and repetitive architectural features that lead to perceptual aliasing. This report details how a combination of foundation models and geometric optimization can overcome these hurdles.
Methodology
Method I: Dense Sequential Matching
The baseline approach, referred to as Method I, focuses on robust sequential matching using RoMa (Robust Matcher). Unlike traditional feature-based methods that rely on local descriptors like SIFT, RoMa utilizes a deep neural network to predict a dense warp field between two images. This field maps pixels from one image to their corresponding locations in another, accompanied by a confidence score.
By sampling high-confidence point correspondences from this dense field, the system can estimate relative motion even when visual overlap is low. The final pose estimation is refined using Graph-Cut RANSAC, which provides robustness against outliers in the sampled points.
Method II: SfM Reconstruction with Global Retrieval
The primary contribution and winning strategy (Method II) involves a more holistic reconstruction of the scene using COLMAP. This method moves beyond simple pair-wise matching to build a consistent 3D map of the sequence.
- Keypoint Detection: We utilized DeDoDe (Detect, Don’t Describe), a modern detector that prioritizes repeatability over traditional descriptor matching.
- Guided Matching: To find correspondences between DeDoDe keypoints, we used RoMa’s dense warps. This "dense-to-sparse" guidance combines the global context of dense matching with the sub-pixel precision of sparse keypoints.
- Loop Closure via DINOv2: To handle temporal discontinuities, we used DINOv2 for image retrieval. By identifying non-consecutive images that depict the same scene, we created "loop closures" that link disparate parts of the sequence, effectively acting as a "glue" for the 3D reconstruction.
- Multicrop Strategy: To maximize the number of valid matches in low-overlap scenarios, we performed matching on multiple crops of each image pair, ensuring that even peripheral visual information was utilized.
Results and Performance
The proposed methods were evaluated based on rotation error (milliradians) and translation error (meters). The results demonstrate that the integrated SfM approach (Method II-A) significantly outperforms the sequential baseline.
| Method | Rotation Error (mrad) | Translation Error (m) | Description |
|---|---|---|---|
| Method II-A | 34 | 1.5 | Automated DINOv2 + COLMAP |
| Method I | 39 | 2.0 | Sequential RoMa Baseline |
A key finding was the zero-shot generalization of the deep matchers. Despite being pre-trained on the MegaDepth dataset (comprising internet photos of landmarks), the models performed exceptionally well on the car-mounted urban data of Singapore without any fine-tuning.
Discussion and Conclusion
The success of this approach highlights a shift in computer vision: the integration of large-scale foundation models into classical geometric pipelines. Using dense matchers to guide sparse keypoints provides a robust solution to the low-overlap problem, while self-supervised retrieval models like DINOv2 solve the connectivity issues inherent in sparse sequences.
However, limitations remain. The computational cost of Method II is high, requiring several hours for full sequence processing. Additionally, sequences with absolute zero visual overlap (e.g., extremely sharp turns) still require auxiliary sensor data like IMU or GPS. Future work will explore more efficient matching strategies and multi-modal fusion to create a truly universal localization system.
Summary updated