Fusing Convolutional Neural Network and Geometric Constraint for Image-based Indoor Localization

Jingwei Song, Mitesh Patel, Maani Ghaffari

Abstract

This paper proposes a new image-based localization framework that explicitly localizes the camera/robot by fusing Convolutional Neural Network (CNN) and sequential images' geometric constraints. The camera is localized using a single or few observed images and training images with 6-degree-of-freedom pose labels. A Siamese network structure is adopted to train an image descriptor network, and the visually similar candidate image in the training set is retrieved to localize the testing image geometrically. Meanwhile, a probabilistic motion model predicts the pose based on a constant velocity assumption. The two estimated poses are finally fused using their uncertainties to yield an accurate pose prediction. This method leverages the geometric uncertainty and is applicable in indoor scenarios predominated by diffuse illumination. Experiments on simulation and real data sets demonstrate the efficiency of our proposed method. The results further show that combining the CNN-based framework with geometric constraint achieves better accuracy when compared with CNN-only methods, especially when the training data size is small.

Introduction

Indoor localization—determining a robot's or camera's exact 6-degree-of-freedom (6-DoF) pose—is a cornerstone of autonomous navigation. Current approaches generally fall into two categories: monocular SLAM and deep learning-based regression. While SLAM is effective for local tracking, it often suffers from scale drift and requires continuous image sequences. Conversely, deep learning models like PoseNet can map images directly to poses but act as "black boxes" requiring massive labeled datasets and struggling with novel viewing angles.

This research introduces a hybrid framework that combines deep learning for robust image retrieval with classical geometry for precise pose estimation. By focusing on similarity retrieval rather than direct coordinate regression, the system achieves high accuracy with significantly less training data.

Proposed Methodology

The framework integrates five core modules to provide a robust pose estimate by fusing visual descriptors with physical constraints.

Overall framework of the proposed image-based localization system, integrating CNN-based retrieval with geometric constraints and a motion model.

1. Image Descriptor Network

A Siamese network architecture, based on GoogLeNet, is employed to learn robust image descriptors. Unlike regression-based networks, this model is trained using a contrastive loss to identify the most visually similar training image for any given test frame.

The Siamese network architecture used for training the image descriptor network, featuring Inception modules and shared weights.

2. Geometric Locator and Keyframe Selection

Once a similar training image is retrieved, the system selects co-visible keyframes that share visual features. The Geometric Locator then utilizes forward-intersection to construct a local 3D map and backward-intersection to calculate the test camera's pose by minimizing re-projection errors.

3. Probabilistic Motion Model and Fusion

To mitigate the impact of noisy frames or motion blur, a probabilistic motion model predicts the camera's position based on a constant velocity assumption. The final pose is a weighted fusion of the geometric estimate and the motion model prediction, where weights are dynamically adjusted based on the calculated uncertainty (e.g., re-projection error) of each source.

Experimental Results

The method was evaluated using synthetic data from NVIDIA’s Isaac Sim and real-world benchmarks including Microsoft’s 7Scenes and the TUM RGB-D dataset.

  • Superior Accuracy: On the TUM "desk2" sequence, the proposed method achieved a median error of 0.03m and 2.74°, significantly outperforming PoseNet (0.21m / 15.81°) and ContextualNet (0.05m / 10.75°).
  • Robustness to Lighting: In simulated warehouse environments with challenging lighting, the fusion method maintained a median error of 0.51m, whereas PoseNet's error increased to 3.58m.
  • Data Efficiency: The system demonstrated superior performance even when training data was reduced to 12% of the original set, showcasing its ability to generalize from sparse labels.

Conclusion

By leveraging geometry as a "sanity check" for neural network predictions, this framework provides a more reliable and data-efficient solution for indoor localization. Future work will focus on improving robustness to drastic illumination changes and extending the system to handle uncalibrated cameras with unknown intrinsic parameters.

Summary updated

Abstract

This paper proposes a new image-based localization framework that explicitly localizes the camera/robot by fusing Convolutional Neural Network (CNN) and sequential images' geometric constraints. The camera is localized using a single or few observed images and training images with 6-degree-of-freedom pose labels. A Siamese network structure is adopted to train an image descriptor network, and the visually similar candidate image in the training set is retrieved to localize the testing image geometrically. Meanwhile, a probabilistic motion model predicts the pose based on a constant velocity assumption. The two estimated poses are finally fused using their uncertainties to yield an accurate pose prediction. This method leverages the geometric uncertainty and is applicable in indoor scenarios predominated by diffuse illumination. Experiments on simulation and real data sets demonstrate the efficiency of our proposed method. The results further show that combining the CNN-based framework with geometric constraint achieves better accuracy when compared with CNN-only methods, especially when the training data size is small.
BibTeX
@article{song2022fusing,
  author = {Jingwei Song and Mitesh Patel and Maani Ghaffari},
  title = {{Fusing Convolutional Neural Network and Geometric Constraint for Image-based Indoor Localization}},
  journal = {IEEE RA-L},
  year = {2022},
  doi = {10.1109/lra.2022.3140832},
}
Figure