MambaLoc: Efficient Camera Localisation via State Space Model

Jialu Wang, Kaichen Zhou, Andrew Markham, Niki TrigoniarXiv 2024 · OpenReview

Abstract

Location information is pivotal for the automation and intelligence of terminal devices and edge-cloud IoT systems, such as autonomous vehicles and augmented reality. However, achieving reliable positioning across diverse IoT applications remains challenging due to significant training costs and the necessity of densely collected data. To tackle these issues, we have innovatively applied the selective state space (SSM) model to visual localization, introducing a new model named MambaLoc. The proposed model demonstrates exceptional training efficiency by capitalizing on the SSM model's strengths in efficient feature extraction, rapid computation, and memory optimization, and it further ensures robustness in sparse data environments due to its parameter sparsity. Additionally, we propose the Global Information Selector (GIS), which leverages selective SSM to implicitly achieve the efficient global feature extraction capabilities of Non-local Neural Networks. This design leverages the computational efficiency of the SSM model alongside the Non-local Neural Networks' capacity to capture long-range dependencies with minimal layers. Consequently, the GIS enables effective global information capture while significantly accelerating convergence. Our extensive experimental validation using public indoor and outdoor datasets first demonstrates our model's effectiveness, followed by evidence of its versatility with various existing localization models.

Introduction

Visual localization—the task of determining a camera's 6-Degree-of-Freedom (6-DoF) pose from a single image—is a cornerstone of autonomous driving and augmented reality. Currently, researchers face a trade-off between structure-based methods, which are accurate but computationally heavy, and Absolute Pose Regression (APR) methods, which are lightweight during inference but slow to train and prone to overfitting in sparse data environments.

MambaLoc addresses these challenges by leveraging State Space Models (SSMs) to achieve state-of-the-art camera pose estimation with significantly reduced training time. By introducing a "Global Information Selector" (GIS), the framework enables efficient global feature extraction with linear complexity ($O(N)$), contrasting with the quadratic complexity ($O(N^2)$) of Transformers.

Methodology

The MambaLoc architecture integrates the Mamba selective State Space Model into the localization pipeline to provide global context efficiently.

MambaLoc Architecture and GIS Module

Architecture and Global Information Selector (GIS)

The model utilizes a shared CNN backbone for initial feature extraction, followed by separate branches for position (translation) and orientation (rotation). Each branch incorporates a Transformer-Encoder and the proposed Global Information Selector (GIS).

The GIS is the core innovation, utilizing a bidirectional Mamba module to selectively compress visual features. Unlike standard attention mechanisms that compare every pixel to every other pixel, GIS employs Mamba’s hardware-aware selective scan to capture long-range dependencies. This allows it to function as a "Non-local Neural Network" with substantially lower overhead. To handle the spatial nature of images, the GIS processes feature sequences in both forward and backward directions, ensuring comprehensive spatial context.

Hybrid Knowledge Distillation

For deployment on low-power terminal devices, the authors distilled the "teacher" MambaLoc model into a lightweight EfficientNet-B0 "student." This process uses a combination of feature-based and logit-based loss functions to maintain performance while reducing model size by approximately 47.6%.

Experimental Results

MambaLoc was evaluated on the 7Scenes (indoor) and Cambridge Landmarks (outdoor) benchmarks.

Training Efficiency and Accuracy

A key finding is that MambaLoc achieves SOTA-level accuracy in a single training phase without requiring secondary fine-tuning. On the 7Scenes dataset, MambaLoc reduced average training time to just 14 minutes, a massive improvement over TransPoseNet (178 minutes) and PoseNet (197 minutes).

Performance Comparison on 7Scenes Dataset

As shown in the table above, MambaLoc achieves an average translation error of 0.17m on 7Scenes, matching or exceeding the performance of existing APR methods while being significantly faster to train.

Robustness to Data Sparsity

MambaLoc demonstrates remarkable robustness in environments with extremely sparse training data. In experiments on the "Heads" scene, it maintained competitive accuracy using only 0.05% of the training data, completing training in under 23 seconds. This suggests that SSMs are less prone to the memorization issues typical of Transformers in sparse-data regimes. Furthermore, the GIS module proved to be a "universal booster"; when integrated into older architectures like PoseNet, it reduced training times by up to 88% while improving accuracy.

Conclusion

MambaLoc presents a significant step forward in efficient visual localization. By replacing or augmenting traditional attention mechanisms with selective SSMs, it achieves high accuracy with linear computational complexity. Future work will explore its potential for loop closure detection in SLAM systems and its performance in more dynamic outdoor environments. While highly efficient, the model's training speed benefits are most pronounced on hardware supporting optimized SSM CUDA kernels.

Summary updated

Abstract

Location information is pivotal for the automation and intelligence of terminal devices and edge-cloud IoT systems, such as autonomous vehicles and augmented reality. However, achieving reliable positioning across diverse IoT applications remains challenging due to significant training costs and the necessity of densely collected data. To tackle these issues, we have innovatively applied the selective state space (SSM) model to visual localization, introducing a new model named MambaLoc. The proposed model demonstrates exceptional training efficiency by capitalizing on the SSM model's strengths in efficient feature extraction, rapid computation, and memory optimization, and it further ensures robustness in sparse data environments due to its parameter sparsity. Additionally, we propose the Global Information Selector (GIS), which leverages selective SSM to implicitly achieve the efficient global feature extraction capabilities of Non-local Neural Networks. This design leverages the computational efficiency of the SSM model alongside the Non-local Neural Networks' capacity to capture long-range dependencies with minimal layers. Consequently, the GIS enables effective global information capture while significantly accelerating convergence. Our extensive experimental validation using public indoor and outdoor datasets first demonstrates our model's effectiveness, followed by evidence of its versatility with various existing localization models. Our code and models are publicly available to support further research and development in this area.
BibTeX
@misc{wang2024mambaloc,
  author = {Jialu Wang and Kaichen Zhou and Andrew Markham and Niki Trigoni},
  title = {{MambaLoc: Efficient Camera Localisation via State Space Model}},
  year = {2024},
  eprint = {2408.09680},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.09680},
}
Figure