From Sparse to Dense: Camera Relocalization with Scene-Specific Detector from Feature Gaussian Splatting

Zhiwei Huang, Hailin Yu, Yichun Shentu, Jin Yuan, Guofeng Zhang

Abstract

Camera relocalization is essential for autonomous systems, yet existing methods often struggle with textureless environments or require significant pose priors. We present STDLoc, a novel relocalization pipeline that utilizes Feature Gaussian Splatting as a scene representation. By distilling local features into a 3D Gaussian field, we enable a sparse-to-dense localization paradigm. Our approach introduces a matching-oriented Gaussian sampling strategy and a scene-specific detector for robust initial pose estimation, followed by dense feature map alignment for high-precision refinement. Experimental results on the 7-Scenes and Cambridge Landmarks datasets demonstrate that STDLoc achieves state-of-the-art accuracy and recall without requiring image retrieval or initial pose priors.

Introduction

Camera relocalization—estimating the 6-degree-of-freedom (6DoF) pose of a camera from a single image—is a fundamental challenge in robotics and augmented reality. Traditional structure-based methods rely on sparse point clouds, which often fail in environments with repetitive or weak textures. Conversely, absolute pose regression (APR) methods offer simplicity but lack the precision required for many applications and struggle with large-scale outdoor scenes.

Recent advances in radiance fields, such as 3D Gaussian Splatting (3DGS), have shown potential, but most existing implementations require a coarse initial pose to begin optimization. STDLoc addresses these limitations by providing a full, robust pipeline that operates without prior pose information, bridging the gap between sparse landmark matching and dense volumetric representation.

Methodology

The STDLoc architecture is built upon four primary innovations that facilitate a transition from sparse initial estimation to dense refinement:

1. Feature Gaussian Splatting

Beyond storing color and geometry, each 3D Gaussian in the scene incorporates a feature vector distilled from a general-purpose feature extractor (such as SuperPoint). This creates a renderable "Feature Field" that allows the system to synthesize feature maps from any perspective.

2. Matching-Oriented Sampling

To manage computational complexity, the system selects a subset of "landmarks" from millions of Gaussians. These landmarks are chosen based on their spatial distribution and their ability to be recognized consistently across multiple viewpoints, ensuring a reliable set of 3D points for matching.

3. Scene-Specific Detector

Rather than relying on generic keypoint detectors that look for generic interest points, STDLoc employs a lightweight, specialized CNN trained for each specific scene. This detector is optimized to identify 2D locations in a query image that correspond directly to the pre-selected 3D landmarks in the Gaussian map.

4. Sparse-to-Dense Pipeline

  • Sparse Stage: The scene-specific detector identifies landmarks to calculate a robust initial pose via Perspective-n-Point (PnP). This stage eliminates the need for the image retrieval step common in other coarse-to-fine pipelines.
  • Dense Stage: Using the initial pose, the system renders a dense feature map from the 3D model and aligns it with the query image's feature map. This dense-to-dense matching provides the final, high-precision pose refinement.

Experimental Evaluation

STDLoc was evaluated on two benchmark datasets: 7-Scenes (indoor) and Cambridge Landmarks (outdoor).

  • Superior Accuracy: On the 7-Scenes dataset, STDLoc reduced translation and rotation errors by approximately 30% compared to previous state-of-the-art methods like ACE and DSAC*.
  • High Recall: The system achieved a 99.1% recall at the strict 5cm/5° threshold for indoor scenes, demonstrating exceptional reliability.
  • Outdoor Robustness: In outdoor environments, STDLoc outperformed other radiance-field-based methods, particularly in rotation accuracy, despite challenges like lighting changes and dynamic objects.
  • Efficiency: The pipeline operates at approximately 7 FPS on an NVIDIA RTX 4090, making it viable for real-time applications where high precision is required.

Discussion and Limitations

The success of STDLoc highlights the value of feature distillation and scene-specific optimization. By bypassing the traditional image retrieval bottleneck, the system achieves more direct 2D-3D matching. However, several areas for improvement remain:

  • Scale Limits: Training on city-scale scenes is currently constrained by memory limits, potentially requiring "chunked" training or Level of Detail (LoD) approaches in the future.
  • Artifacts: "Floaters"—common artifacts in Gaussian Splatting—can occasionally occlude rendered feature maps and degrade the dense matching stage.
  • Training Time: While inference is fast, the initial mapping process (approximately 90 minutes for Feature Gaussians and 30 minutes for the detector) remains a target for future optimization.

Conclusion

STDLoc represents a significant step forward in camera relocalization. By leveraging the representational power of Feature Gaussian Splatting and a scene-specific detection strategy, it provides a robust, high-precision solution that excels in both indoor and outdoor environments without the need for initial pose priors.

Summary updated

Abstract

This paper presents a novel camera relocalization method, STDLoc, which leverages Feature Gaussian as scene representation. STDLoc is a full relocalization pipeline that can achieve accurate relocalization without relying on any pose prior. Unlike previous coarse-to-fine localization methods that require image retrieval first and then feature matching, we propose a novel sparse-to-dense localization paradigm. Based on this scene representation, we introduce a novel matching-oriented Gaussian sampling strategy and a scene-specific detector to achieve efficient and robust initial pose estimation. Furthermore, based on the initial localization results, we align the query feature map to the Gaussian feature field by dense feature matching to enable accurate localization. The experiments on indoor and outdoor datasets show that STDLoc outperforms current state-of-the-art localization methods in terms of localization accuracy and recall.
BibTeX
@inproceedings{huang2025from,
  author = {Zhiwei Huang and Hailin Yu and Yichun Shentu and Jin Yuan and Guofeng Zhang},
  title = {{From Sparse to Dense: Camera Relocalization with Scene-Specific Detector from Feature Gaussian Splatting}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=OZtIkwU4TY},
}
Figure