Scale-Robust Localization Using General Object Landmarks

Andrew Holliday, Gregory Dudek

Introduction

Visual localization—the process by which a robot determines its pose based on camera imagery—is a fundamental requirement for autonomous navigation. However, existing methods frequently struggle with significant variations in visual scale. For instance, a ground robot attempting to localize using a map generated by a high-altitude drone faces a substantial "scale gap." While classical techniques like the Scale-Invariant Feature Transform (SIFT) are theoretically scale-invariant, they typically fail when the scale difference between the map and the query image exceeds a factor of 3×. This limitation hinders loop closure over long distances and prevents interoperability between multi-robot systems operating at different altitudes.

Methodology

This paper proposes a hybrid, coarse-to-fine matching pipeline that requires no environment-specific training. The system leverages the high-level robustness of deep learning to guide the precision of classical computer vision.

  1. Object Proposal: The system utilizes Selective Search to identify potential object regions within an image pair. These regions are treated as class-agnostic physical entities rather than specific semantic categories.
  2. Deep Description: Each proposed region is processed through a ResNet-50 backbone pre-trained on ImageNet. Features are extracted from intermediate layers, providing high-level abstractions that are more robust to scale changes than raw pixel data.
  3. Object Matching: Regions are matched across images using cosine distance and cross-checking to establish "anchor" regions representing the same physical object at different scales.
  4. Guided SIFT Matching: SIFT features are extracted exclusively within these matched bounding boxes. This targeted approach filters out background noise and prevents the combinatorial explosion of false matches that typically occurs at high scale differences.
  5. Geometric Verification: Finally, the matched points are used to estimate a homography or essential matrix via RANSAC to determine the relative pose.
The proposed pipeline uses Selective Search for object proposals and extracts features from intermediate ResNet layers to establish robust region matches.

Experimental Evaluation

The system was evaluated on two datasets: the KITTI Odometry Benchmark, where frames were subsampled to create artificial scale gaps, and the Montreal Image Pair Dataset, which contains urban scene pairs with scale differences ranging from 1.5× to 15×.

Key Results

The hybrid approach demonstrated significant improvements over baseline methods:

  • Reduced Failure Rate: On the KITTI dataset, the proposed method reduced the localization failure rate by 58% compared to SIFT alone.
  • Improved Accuracy: The system achieved a 43% improvement in rotational error ($r_{err}$) over the baseline.
  • Extreme Scale Robustness: In the Montreal dataset, the system successfully determined accurate homographies at scale factors of 6× and 7×, conditions under which standard SIFT matching failed entirely.
Comparison of localization under extreme scale change. While standard SIFT fails to find a valid match (bottom left), the proposed method using intermediate ResNet features (res3d) successfully localizes the object (bottom right).

Discussion and Insights

A critical finding of this research is that deep learning and classical computer vision are highly complementary for localization. CNNs provide the high-level "what" and "roughly where" (robust to scale), while SIFT provides the low-level "exactly where" necessary for metric localization. Furthermore, the authors found that intermediate layers (such as res3d or res4f) often outperformed the deepest layers, suggesting a "sweet spot" of abstraction for spatial tasks.

Because the system is class-agnostic, it can be deployed "out-of-the-box" in diverse environments without retraining. Future work will focus on extending this image-pair matching to full global localization against massive databases and exploring multi-layer descriptors to further enhance scale invariance.

Summary updated

Abstract

Visual localization under large changes in scale is an important capability in many robotic mapping applications, such as localizing at low altitudes in maps built at high altitudes, or performing loop closure over long distances. Existing approaches, however, are robust only up to about a 3× difference in scale between map and query images. We propose a novel combination of deep-learning-based object features and state-of-the-art SIFT point-features that yields improved robustness to scale change. This technique is training-free and class-agnostic, and in principle can be deployed in any environment out-of-the-box. We evaluate the proposed technique on the KITTI Odometry benchmark and on a novel dataset of outdoor images exhibiting changes in visual scale of 7× and greater, which we have released to the public. Our technique consistently outperforms localization using either SIFT features or the proposed object features alone, achieving both greater accuracy and much lower failure rates under large changes in scale.
BibTeX
@inproceedings{holliday2018scale,
  author = {Andrew Holliday and Gregory Dudek},
  title = {{Scale-Robust Localization Using General Object Landmarks}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  doi = {10.1109/iros.2018.8594011},
}
Figure