Scene-agnostic Pose Regression for Visual Localization

Junwei Zheng, Ruiping Liu, Yufan Chen, Zhenfang Chen, Kailun Yang, Jiaming Zhang, Rainer Stiefelhagen

Abstract

Visual localization is a fundamental challenge in robotics, typically requiring scene-specific retraining or extensive image databases. This paper proposes Scene-agnostic Pose Regression (SPR), a paradigm shift that enables robots to localize in unseen environments by predicting poses relative to a trajectory's origin. We introduce SPR-Mamba, a dual-branch architecture utilizing State Space Models for long-range dependency modeling, and 360SPR, a large-scale panoramic dataset. Our approach achieves drift-free localization in unknown scenes, significantly outperforming traditional absolute and relative pose regression methods.

Introduction

Visual localization—the process of determining a device's position and orientation from images—is critical for autonomous navigation. Existing methods generally fall into three categories, each with significant drawbacks:

  1. Absolute Pose Regression (APR): Requires retraining for every new scene.
  2. Relative Pose Regression (RPR): Generalizes better but depends on a large, computationally expensive image "atlas."
  3. Visual Odometry (VO): Works anywhere but suffers from "drift," where errors accumulate over time.
Overview of the SPR framework, illustrating trajectories for different robot types and the transition from known to unknown environments.

To resolve this "localization dilemma," the authors introduce the SPR task. Instead of mapping to a global coordinate system, the model uses the first image of a sequence ($I_1$) as the origin and predicts the 6D pose of subsequent query images ($I_q$) relative to that origin. This allows the model to learn the geometry of movement rather than specific landmark appearances.

Methodology: SPR-Mamba

The proposed SPR-Mamba model is designed to handle long trajectories with high efficiency. It consists of several key components:

  • Feature Extraction: A frozen DINO (Vision Transformer) backbone extracts robust visual features.
  • Local Branch: Captures short-term, frame-to-frame motion cues.
  • Global Branch: Employs Mamba (State Space Models) to aggregate information across entire sequences. Unlike standard Transformers, Mamba maintains linear complexity relative to sequence length, making it ideal for the long-range dependencies found in robotic paths.
  • Fusion and Prediction: The model fuses local and global information to regress the final translation and rotation. By anchoring predictions to the initial frame, the global branch effectively eliminates the cumulative drift common in VO.

The 360SPR Dataset

A significant contribution of this work is the creation of the 360SPR dataset, designed to address the limitations of existing benchmarks.

  • Scale: It contains over 200,000 photorealistic panoramas and 3.6 million pinhole images across 270 diverse indoor environments.
  • Multi-Height Sensing: To ensure the model works across different robotic platforms, data was collected at three distinct heights: 0.1m (e.g., vacuum robots), 0.5m (e.g., quadrupeds), and 1.7m (e.g., humanoids).
Sensor heights used in the 360SPR dataset, simulating vacuum robots, quadrupeds, and humanoids.

The use of 360° panoramas is central to the SPR task, as the wider field of view provides the necessary visual overlap to maintain stable relative pose estimation across varying viewpoints.

Experimental Results

SPR-Mamba demonstrates state-of-the-art performance in zero-shot generalization to unknown scenes. In unseen environments, SPR-Mamba reduced translation and rotation errors by over 23m and 43° compared to APR, and 7m and 16° compared to RPR.

On the 360Loc benchmark, the model achieved a median error of 1.94m and 1.44° in entirely new environments, proving its efficacy on real photographic data. Ablation studies confirmed that the Global Branch is the "secret sauce"; without it, the model loses its anchor to the starting point, leading to significant performance degradation over long trajectories.

Conclusion and Future Work

SPR-Mamba represents a significant step toward general-purpose robot navigation. By disentangling scene appearance from motion geometry, the researchers enable localization in novel environments without the need for maps or retraining. Future work will focus on handling dynamic obstacles in crowded environments and explicitly modeling the distortions inherent in panoramic imagery to further refine precision.

Summary updated

Abstract

Absolute Pose Regression (APR) predicts 6D camera poses but lacks the adaptability to unknown environments without retraining, while Relative Pose Regression (RPR) generalizes better yet requires a large image retrieval database. Visual Odometry (VO) generalizes well in unseen environments but suffers from accumulated error in open trajectories. To address this dilemma, we introduce a new task, Scene-agnostic Pose Regression (SPR), which can achieve accurate pose regression in a flexible way while eliminating the need for retraining or databases. To benchmark SPR, we created a large-scale dataset, 360SPR, with over 200K photorealistic panoramas, 3.6M pinhole images and camera poses in 270 scenes at three different sensor heights. Furthermore, a SPR-Mamba model is initially proposed to address SPR in a dual-branch manner. Extensive experiments and studies demonstrate the effectiveness of our SPR paradigm, dataset, and model. In the unknown scenes of both 360SPR and 360Loc datasets, our method consistently outperforms APR, RPR and VO. The dataset and code are available at SPR.
BibTeX
@inproceedings{zheng2025scene,
  author = {Junwei Zheng and Ruiping Liu and Yufan Chen and Zhenfang Chen and Kailun Yang and Jiaming Zhang and Rainer Stiefelhagen},
  title = {{Scene-agnostic Pose Regression for Visual Localization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02523},
}
Figure