CamNet: Coarse-to-Fine Retrieval for Camera Re-Localization

Mingyu Ding, Zhe Wang, Jiankai Sun, Jianping Shi, Ping Luo

Introduction

Camera re-localization—the task of determining the 6-Degree-of-Freedom (6-DoF) pose of a camera from a single image—is fundamental to robotics, augmented reality, and autonomous navigation. While 3D-based methods offer high precision, they often struggle with scalability and computational overhead. Conversely, image-based methods, which either regress absolute poses or utilize image retrieval, frequently encounter a "learning conflict." This conflict arises because the visual features optimal for global scene retrieval (which should be invariant to small viewpoint shifts) differ significantly from those required for precise relative pose regression (which must be sensitive to subtle geometric changes).

CamNet addresses this challenge by introducing a coarse-to-fine retrieval framework that decouples these tasks while maintaining an efficient shared-encoder architecture.

The CamNet Framework

The core of CamNet is a Siamese-style network that utilizes a shared ResNet34 encoder to extract features, which are then processed by three specialized modules. This hierarchical approach allows the system to transition from broad localization to millimeter-level precision.

The CamNet architecture featuring a shared encoder and three specialized modules: ICR, PFR, and PRP.
  1. Image-based Coarse Retrieval (ICR): This module identifies a set of candidate "anchor" images from a database. To optimize retrieval, the authors introduced a Bilateral Frustum Loss, which calculates the 3D overlap of camera viewing cones from both perspectives. This is supplemented by an Angle-based Loss and a Hard Triplet Loss to ensure the network can distinguish between visually similar but geographically distant locations.
  2. Pose-based Fine Retrieval (PFR): After initial retrieval, this module performs a rapid, coarse relative pose estimation. This estimate is used to re-query the database, selecting a reference image that is geometrically closer to the query than the initial visual match.
  3. Precise Relative Pose Regression (PRP): Finally, the PRP module calculates the high-precision 6-DoF offset between the query image and the refined reference image. By separating the decoders for these tasks, CamNet resolves the feature conflict inherent in single-stage methods.

Experimental Results

CamNet was evaluated across diverse environments, including the indoor 7-Scenes dataset, the large-scale outdoor Oxford RobotCar dataset, and the ApolloScape city-view dataset.

In indoor settings, CamNet achieved a median error of 0.04m and 1.69° on 7-Scenes, a massive improvement over the previous state-of-the-art (MapNet++ at 0.21m/7.77°). In outdoor scenarios, the performance gap was even more pronounced; on the Oxford RobotCar "LOOP" sequence, CamNet reduced the mean translation error to 2.56m, compared to 11.37m for NN-Net.

Qualitative comparison of estimated trajectories (red) against ground truth (green) across various methods.

As shown in the trajectory visualizations, CamNet (Ours) produces paths that closely align with the ground truth (green), whereas baseline methods like PoseNet and MapNet++ exhibit significant drift and inaccuracies.

Key Insights

  • Task Decoupling: Separating retrieval and regression objectives while sharing a backbone allows for specialized feature extraction without compromising efficiency.
  • Two-Stage Retrieval: Using an initial pose estimate to "jump" to a better reference image significantly simplifies the final regression task.
  • Loss Function Optimization: The researchers found that the $L_1$ norm is more robust for pose regression than the standard $L_2$ norm, and that bilateral frustum overlap provides a superior training signal for retrieval.

Conclusion and Future Work

CamNet sets a new benchmark for image-based camera re-localization by effectively bridging the gap between retrieval and regression. While the method currently depends on a pre-built image database, its ability to generalize to new environments—as demonstrated on ApolloScape—makes it a powerful tool for large-scale autonomous systems. Future research will likely focus on improving the scalability and search efficiency of the retrieval process for even larger datasets.

Summary updated

Abstract

Camera re-localization is an important but challenging task in applications like robotics and autonomous driving. Recently, retrieval-based methods have been considered as a promising direction as they can be easily generalized to novel scenes. Despite significant progress has been made, we observe that the performance bottleneck of previous methods actually lies in the retrieval module. These methods use the same features for both retrieval and relative pose regression tasks which have potential conflicts in learning. To this end, here we present a coarse-to-fine retrieval-based deep learning framework, which includes three steps, i.e., image-based coarse retrieval, pose-based fine retrieval and precise relative pose regression. With our carefully designed retrieval module, the relative pose regression task can be surprisingly simpler. We design novel retrieval losses with batch hard sampling criterion and two-stage retrieval to locate samples that adapt to the relative pose regression task. Extensive experiments show that our model (CamNet) outperforms the state-of-the-art methods by a large margin on both indoor and outdoor datasets.
BibTeX
@inproceedings{ding2019camnet,
  author = {Mingyu Ding and Zhe Wang and Jiankai Sun and Jianping Shi and Ping Luo},
  title = {{CamNet: Coarse-to-Fine Retrieval for Camera Re-Localization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00296},
}
Figure