Leveraging Cutting Edge Deep Learning Based Image Matching for Reconstructing a Large Scene from Sparse Images

Georg Bökman, Johan EdstedtarXiv 2023 · DOI · arXiv

Abstract

This paper presents the methodology behind the winning entry for the AISG–SLA Visual Localisation Challenge at IJCAI 2023. The core challenge involved reconstructing large-scale urban scenes from sparse image sequences characterized by low overlap and temporal discontinuities. Our approach integrates state-of-the-art deep learning matchers—specifically RoMa and DeDoDe—with the robust Structure-from-Motion (SfM) framework, COLMAP. By employing DINOv2 for global image retrieval and a dense-to-sparse guided matching strategy, we successfully bridged gaps in sequential data, achieving state-of-the-art localization accuracy in complex urban environments.

Introduction

Visual localization—the process of estimating a camera's pose within a known environment—is a fundamental task for autonomous navigation and robotics. However, real-world datasets often present significant challenges that break traditional sequential pipelines. In the AISG-SLA Visual Localisation Challenge, researchers were tasked with processing image sequences from Singapore's urban landscape. These sequences featured low-frequency captures (resulting in minimal overlap between consecutive frames), "time jumps" where data was missing, and repetitive architectural features that lead to perceptual aliasing. This report details how a combination of foundation models and geometric optimization can overcome these hurdles.

Methodology

Method I: Dense Sequential Matching

The baseline approach, referred to as Method I, focuses on robust sequential matching using RoMa (Robust Matcher). Unlike traditional feature-based methods that rely on local descriptors like SIFT, RoMa utilizes a deep neural network to predict a dense warp field between two images. This field maps pixels from one image to their corresponding locations in another, accompanied by a confidence score.

Overview of Method I: Sequential matching using RoMa dense warps and Graph-Cut RANSAC.

By sampling high-confidence point correspondences from this dense field, the system can estimate relative motion even when visual overlap is low. The final pose estimation is refined using Graph-Cut RANSAC, which provides robustness against outliers in the sampled points.

Method II: SfM Reconstruction with Global Retrieval

The primary contribution and winning strategy (Method II) involves a more holistic reconstruction of the scene using COLMAP. This method moves beyond simple pair-wise matching to build a consistent 3D map of the sequence.

  1. Keypoint Detection: We utilized DeDoDe (Detect, Don’t Describe), a modern detector that prioritizes repeatability over traditional descriptor matching.
  2. Guided Matching: To find correspondences between DeDoDe keypoints, we used RoMa’s dense warps. This "dense-to-sparse" guidance combines the global context of dense matching with the sub-pixel precision of sparse keypoints.
  3. Loop Closure via DINOv2: To handle temporal discontinuities, we used DINOv2 for image retrieval. By identifying non-consecutive images that depict the same scene, we created "loop closures" that link disparate parts of the sequence, effectively acting as a "glue" for the 3D reconstruction.
  4. Multicrop Strategy: To maximize the number of valid matches in low-overlap scenarios, we performed matching on multiple crops of each image pair, ensuring that even peripheral visual information was utilized.
Visualization of point correspondences established between sparse urban images.

Results and Performance

The proposed methods were evaluated based on rotation error (milliradians) and translation error (meters). The results demonstrate that the integrated SfM approach (Method II-A) significantly outperforms the sequential baseline.

Method Rotation Error (mrad) Translation Error (m) Description
Method II-A 34 1.5 Automated DINOv2 + COLMAP
Method I 39 2.0 Sequential RoMa Baseline

A key finding was the zero-shot generalization of the deep matchers. Despite being pre-trained on the MegaDepth dataset (comprising internet photos of landmarks), the models performed exceptionally well on the car-mounted urban data of Singapore without any fine-tuning.

Discussion and Conclusion

The success of this approach highlights a shift in computer vision: the integration of large-scale foundation models into classical geometric pipelines. Using dense matchers to guide sparse keypoints provides a robust solution to the low-overlap problem, while self-supervised retrieval models like DINOv2 solve the connectivity issues inherent in sparse sequences.

However, limitations remain. The computational cost of Method II is high, requiring several hours for full sequence processing. Additionally, sequences with absolute zero visual overlap (e.g., extremely sharp turns) still require auxiliary sensor data like IMU or GPS. Future work will explore more efficient matching strategies and multi-modal fusion to create a truly universal localization system.

Summary updated

Abstract

We present the top ranked solution for the AISG-SLA Visual Localisation Challenge benchmark (IJCAI 2023), where the task is to estimate relative motion between images taken in sequence by a camera mounted on a car driving through an urban scene. For matching images we use our recent deep learning based matcher RoMa. Matching image pairs sequentially and estimating relative motion from point correspondences sampled by RoMa already gives very competitive results -- third rank on the challenge benchmark. To improve the estimations we extract keypoints in the images, match them using RoMa, and perform structure from motion reconstruction using COLMAP. We choose our recent DeDoDe keypoints for their high repeatability. Further, we address time jumps in the image sequence by matching specific non-consecutive image pairs based on image retrieval with DINOv2. These improvements yield a solution beating all competitors. We further present a loose upper bound on the accuracy obtainable by the image retrieval approach by also matching hand-picked non-consecutive pairs.
BibTeX
@misc{bokman2023leveraging,
  author = {Georg Bökman and Johan Edstedt},
  title = {{Leveraging Cutting Edge Deep Learning Based Image Matching for Reconstructing a Large Scene from Sparse Images}},
  year = {2023},
  eprint = {2310.01092},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.01092},
}
Figure