From Sparse to Dense: Camera Relocalization with Scene-Specific Detector from Feature Gaussian Splatting

Zhiwei Huang, Hailin Yu, Yichun Shentu, Jin Yuan, Guofeng Zhang

Abstract

This paper presents a novel camera relocalization method, STDLoc, which leverages Feature Gaussian as scene representation. STDLoc is a full relocalization pipeline that can achieve accurate relocalization without relying on any pose prior. Unlike previous coarse-to-fine localization methods that require image retrieval first and then feature matching, we propose a novel sparse-to-dense localization paradigm. Based on this scene representation, we introduce a novel matching-oriented Gaussian sampling strategy and a scene-specific detector to achieve efficient and robust initial pose estimation. Furthermore, based on the initial localization results, we align the query feature map to the Gaussian feature field by dense feature matching to enable accurate localization. The experiments on indoor and outdoor datasets show that STDLoc outperforms current state-of-the-art localization methods in terms of localization accuracy and recall.

Introduction

Camera relocalization—the task of determining a camera's exact 6DoF pose relative to a known scene—is a cornerstone of robotics and AR/VR. Historically, researchers have faced a trade-off between different scene representations. Sparse methods (SfM) are fast but fail in textureless environments. Neural methods (APR/SCR) struggle to scale to large outdoor scenes. Radiance field methods (NeRF/3DGS) are excellent for rendering but often rely on photometric alignment, which is sensitive to lighting changes.

STDLoc bridges these gaps by introducing a Sparse-to-Dense localization paradigm built upon Feature Gaussian Splatting. This approach combines the efficiency of sparse landmark matching with the precision of dense feature field alignment.

Overview of the STDLoc sparse-to-dense localization paradigm, showing the transition from landmark-based coarse pose estimation to dense feature field alignment.

Methodology

The core of STDLoc is the Feature Gaussian representation, which augments standard 3D Gaussians with high-dimensional feature vectors distilled from general-purpose extractors.

1. Feature Gaussian Training

Instead of relying solely on RGB color, each 3D Gaussian stores a feature vector. This allows the system to match semantic features rather than raw pixels, significantly improving robustness to environmental changes like lighting or weather. The training process involves optimizing these features alongside geometry through a differentiable rasterization pipeline.

The training process for Feature Gaussians, where features are distilled from a dense extractor and optimized alongside RGB and geometry through rasterization.

2. Sparse-to-Dense Pipeline

The localization process occurs in two distinct stages:

  • Sparse Stage (Coarse Pose): A lightweight, scene-specific CNN detector is trained to identify the 2D locations of a pre-selected set of 3D "landmarks." By matching these landmarks in a query image, the system uses a Perspective-n-Point (PnP) algorithm to estimate a coarse initial pose.
  • Dense Stage (Fine Pose): Using the coarse pose as a starting point, the system renders a dense feature map from the Feature Gaussian field. It then performs dense matching between the query image's features and the rendered map to "snap" the camera into a highly accurate final pose.

Experimental Evaluation

The authors evaluated STDLoc on the 7-Scenes (indoor) and Cambridge Landmarks (outdoor) datasets, comparing it against current state-of-the-art (SOTA) methods.

Key Results

  • Superior Accuracy: On the 7-Scenes benchmark, STDLoc reduced translation and rotation errors by approximately 30% compared to top-performing methods like ACE and DSAC*.
  • High Recall: The system achieved a 99.1% recall at the 5cm/5° threshold on indoor scenes, meaning it almost never fails to find a highly accurate pose.
  • Outdoor Performance: In large-scale outdoor environments, STDLoc consistently outperformed radiance-field-based competitors, particularly in rotation accuracy.
  • Efficiency: Despite its complexity, the system achieves an inference speed of approximately 7 FPS on a single NVIDIA RTX 4090.

Conclusion

STDLoc represents a significant advancement in camera relocalization by effectively combining sparse matching with dense volumetric feature fields. By matching distilled features rather than raw pixels and utilizing a scene-specific detector, it achieves high precision and robustness across diverse environments. While the system requires initial training overhead for each new scene, its performance gains in accuracy and recall set a new benchmark for the field.

Summary updated

Abstract

This paper presents a novel camera relocalization method, STDLoc, which leverages Feature Gaussian as scene representation. STDLoc is a full relocalization pipeline that can achieve accurate relocalization without relying on any pose prior. Unlike previous coarse-to-fine localization methods that require image retrieval first and then feature matching, we propose a novel sparse-to-dense localization paradigm. Based on this scene representation, we introduce a novel matching-oriented Gaussian sampling strategy and a scene-specific detector to achieve efficient and robust initial pose estimation. Furthermore, based on the initial localization results, we align the query feature map to the Gaussian feature field by dense feature matching to enable accurate localization. The experiments on indoor and outdoor datasets show that STDLoc outperforms current state-of-the-art localization methods in terms of localization accuracy and recall. Our code is available on the project website: https://zju3dv.github.io/STDLoc.
BibTeX
@inproceedings{huang2025from,
  author = {Zhiwei Huang and Hailin Yu and Yichun Shentu and Jin Yuan and Guofeng Zhang},
  title = {{From Sparse to Dense: Camera Relocalization with Scene-Specific Detector from Feature Gaussian Splatting}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=OZtIkwU4TY},
}
Figure