Abstract
Camera relocalization is essential for autonomous systems, yet existing methods often struggle with textureless environments or require significant pose priors. We present STDLoc, a novel relocalization pipeline that utilizes Feature Gaussian Splatting as a scene representation. By distilling local features into a 3D Gaussian field, we enable a sparse-to-dense localization paradigm. Our approach introduces a matching-oriented Gaussian sampling strategy and a scene-specific detector for robust initial pose estimation, followed by dense feature map alignment for high-precision refinement. Experimental results on the 7-Scenes and Cambridge Landmarks datasets demonstrate that STDLoc achieves state-of-the-art accuracy and recall without requiring image retrieval or initial pose priors.
Introduction
Camera relocalization—estimating the 6-degree-of-freedom (6DoF) pose of a camera from a single image—is a fundamental challenge in robotics and augmented reality. Traditional structure-based methods rely on sparse point clouds, which often fail in environments with repetitive or weak textures. Conversely, absolute pose regression (APR) methods offer simplicity but lack the precision required for many applications and struggle with large-scale outdoor scenes.
Recent advances in radiance fields, such as 3D Gaussian Splatting (3DGS), have shown potential, but most existing implementations require a coarse initial pose to begin optimization. STDLoc addresses these limitations by providing a full, robust pipeline that operates without prior pose information, bridging the gap between sparse landmark matching and dense volumetric representation.
Methodology
The STDLoc architecture is built upon four primary innovations that facilitate a transition from sparse initial estimation to dense refinement:
1. Feature Gaussian Splatting
Beyond storing color and geometry, each 3D Gaussian in the scene incorporates a feature vector distilled from a general-purpose feature extractor (such as SuperPoint). This creates a renderable "Feature Field" that allows the system to synthesize feature maps from any perspective.
2. Matching-Oriented Sampling
To manage computational complexity, the system selects a subset of "landmarks" from millions of Gaussians. These landmarks are chosen based on their spatial distribution and their ability to be recognized consistently across multiple viewpoints, ensuring a reliable set of 3D points for matching.
3. Scene-Specific Detector
Rather than relying on generic keypoint detectors that look for generic interest points, STDLoc employs a lightweight, specialized CNN trained for each specific scene. This detector is optimized to identify 2D locations in a query image that correspond directly to the pre-selected 3D landmarks in the Gaussian map.
4. Sparse-to-Dense Pipeline
- Sparse Stage: The scene-specific detector identifies landmarks to calculate a robust initial pose via Perspective-n-Point (PnP). This stage eliminates the need for the image retrieval step common in other coarse-to-fine pipelines.
- Dense Stage: Using the initial pose, the system renders a dense feature map from the 3D model and aligns it with the query image's feature map. This dense-to-dense matching provides the final, high-precision pose refinement.
Experimental Evaluation
STDLoc was evaluated on two benchmark datasets: 7-Scenes (indoor) and Cambridge Landmarks (outdoor).
- Superior Accuracy: On the 7-Scenes dataset, STDLoc reduced translation and rotation errors by approximately 30% compared to previous state-of-the-art methods like ACE and DSAC*.
- High Recall: The system achieved a 99.1% recall at the strict 5cm/5° threshold for indoor scenes, demonstrating exceptional reliability.
- Outdoor Robustness: In outdoor environments, STDLoc outperformed other radiance-field-based methods, particularly in rotation accuracy, despite challenges like lighting changes and dynamic objects.
- Efficiency: The pipeline operates at approximately 7 FPS on an NVIDIA RTX 4090, making it viable for real-time applications where high precision is required.
Discussion and Limitations
The success of STDLoc highlights the value of feature distillation and scene-specific optimization. By bypassing the traditional image retrieval bottleneck, the system achieves more direct 2D-3D matching. However, several areas for improvement remain:
- Scale Limits: Training on city-scale scenes is currently constrained by memory limits, potentially requiring "chunked" training or Level of Detail (LoD) approaches in the future.
- Artifacts: "Floaters"—common artifacts in Gaussian Splatting—can occasionally occlude rendered feature maps and degrade the dense matching stage.
- Training Time: While inference is fast, the initial mapping process (approximately 90 minutes for Feature Gaussians and 30 minutes for the detector) remains a target for future optimization.
Conclusion
STDLoc represents a significant step forward in camera relocalization. By leveraging the representational power of Feature Gaussian Splatting and a scene-specific detection strategy, it provides a robust, high-precision solution that excels in both indoor and outdoor environments without the need for initial pose priors.
Summary updated