Abstract
Visual localization is a fundamental capability for autonomous systems, yet existing methods often struggle with a trade-off between generalizability and resource efficiency. Absolute Pose Regression (APR) requires scene-specific retraining, while Relative Pose Regression (RPR) depends on extensive image retrieval databases. Visual Odometry (VO) generalizes well but is plagued by cumulative drift. This paper introduces Scene-agnostic Pose Regression (SPR), a novel paradigm that predicts 6D camera poses relative to a sequence's starting frame, eliminating the need for retraining or external databases. To support this task, we present 360SPR, a large-scale dataset featuring over 200,000 photorealistic panoramas across 270 scenes. We further propose SPR-Mamba, a dual-branch architecture that leverages State Space Models to achieve superior localization accuracy. Our experiments demonstrate that SPR-Mamba consistently outperforms APR, RPR, and VO baselines in unseen environments.
Introduction
Determining a camera's precise position and orientation—visual localization—is critical for robotics and augmented reality. Current methodologies typically fall into three categories, each with significant drawbacks. APR models are fast but environment-specific; they fail in new settings without retraining. RPR models generalize better but require a massive "memory" of reference images to function. VO systems are scene-agnostic but suffer from "drift," where small errors in frame-to-frame tracking accumulate into massive inaccuracies over time.
Scene-agnostic Pose Regression (SPR) bridges these gaps. By defining the first image of any sequence as the coordinate origin (0,0,0), SPR models learn to predict the current pose relative to that specific start point. This approach creates a "local map" on the fly, providing the stability of a reference-based system without the storage overhead of a global database or the drift of incremental odometry.
The 360SPR Dataset
A primary hurdle in developing scene-agnostic models is the lack of diverse, large-scale data. To address this, we developed the 360SPR dataset. Using the Habitat simulator with HM3D and Matterport3D environments, we generated 200,000 photorealistic panoramas and 3.6 million pinhole images across 270 distinct scenes.
Crucially, 360SPR is designed for multi-platform robotics. Data was collected at three distinct sensor heights:
- 0.1m: Representing vacuum and sweeping robots.
- 0.5m: Representing quadrupedal robots.
- 1.7m: Representing humanoid robots and handheld AR devices.
This vertical diversity allows models to learn features that are robust to different perspectives, a common failure point for models trained on single-height datasets.
Methodology: SPR-Mamba
To solve the SPR task, we propose SPR-Mamba, a model that utilizes a dual-branch architecture to process image sequences efficiently. The model uses a frozen DINO backbone for robust visual feature extraction, which are then processed through two distinct paths:
- The Local Branch: This branch analyzes the relationship between adjacent frames. By using auxiliary heads during training, it ensures the model captures short-range motion dynamics accurately.
- The Global Branch: This branch utilizes Mamba blocks (State Space Models) to aggregate information across the entire sequence. Unlike Transformers, which suffer from quadratic computational complexity as sequences grow, Mamba maintains linear complexity. This allows the model to efficiently relate the current query image back to the initial "origin" frame, regardless of the sequence length.
Finally, features from both branches are fused to regress the final 6D pose (3D translation and 3D rotation).
Experimental Results
Extensive benchmarking on the 360SPR and 360Loc datasets reveals several key insights:
- Superior Generalization: In "unseen" scenes (environments not present in the training set), SPR-Mamba reduced translation error by over 7 meters and rotation error by 16° compared to state-of-the-art RPR methods.
- Drift Mitigation: While VO methods showed increasing error as trajectories lengthened, SPR-Mamba’s error remained stable. Because the model always references the starting frame, it effectively "anchors" the trajectory.
- The Panoramic Advantage: Our studies show that a 360° Field of View (FoV) significantly outperforms standard 90° pinhole views. The wider context provided by panoramas prevents the model from losing its orientation in feature-sparse areas.
- Cross-Height Robustness: Models trained on the multi-height 360SPR dataset demonstrated significantly higher accuracy when switching between robot types compared to models trained on single-height data.
Conclusion
SPR-Mamba and the 360SPR dataset represent a significant step toward flexible, scene-agnostic visual localization. By shifting the focus from absolute global coordinates to relative sequence-based coordinates, we provide a path for robots to navigate unknown environments accurately without the need for constant retraining. Future work will focus on addressing spherical distortion in panoramic images and integrating multi-modal Large Language Models (LLMs) to combine geometric precision with semantic scene understanding.
Summary updated