SGLoc: Semantic Localization System for Camera Pose Estimation from 3D Gaussian Splatting Representation

Beining Xu, Siting Zhu, Hesheng Wang

Abstract

Visual localization is a fundamental task in robotics and computer vision, yet traditional methods often struggle in texture-poor environments or require precise initial pose estimates. We present SGLoc, a novel semantic localization system designed for 3D Gaussian Splatting (3DGS) representations. By leveraging a coarse-to-fine strategy that aligns 2D image semantics with 3D scene features, SGLoc achieves robust global localization without prior pose information. Experimental results on the 7Scenes and 12Scenes benchmarks demonstrate state-of-the-art performance, with translation errors as low as 0.15 cm.

1. Introduction

Accurate 6-Degree-of-Freedom (6DoF) camera pose estimation is essential for applications ranging from autonomous navigation to augmented reality. While traditional feature-based methods rely on distinct visual landmarks, they frequently fail in environments with repetitive patterns, low texture, or varying lighting conditions. Conversely, regression-based methods often lack the precision required for high-stakes robotics.

The emergence of 3D Gaussian Splatting (3DGS) has provided a high-fidelity alternative for scene representation. However, most existing 3DGS-based localization frameworks are local in nature, requiring a "pose prior"—a decent initial guess of the camera's location—to initialize the optimization. SGLoc addresses this limitation by introducing a semantic-aware global localization pipeline that operates effectively even when the initial camera position is entirely unknown.

2. Methodology

SGLoc employs a hierarchical, multi-level pose regression strategy to bridge the gap between 2D query images and 3D scene representations.

2.1 Coarse Stage: Semantic Global Retrieval

The first stage focuses on identifying the general vicinity of the camera within a large-scale map. To achieve robustness against lighting and texture variations, the system prioritizes semantic information over raw pixel values.

  • 2D Semantic Extraction: The query image is processed using the Segment Anything Model (SAM) to identify object-level segments. These segments are then encoded into high-dimensional feature vectors using CLIP, creating a semantic "fingerprint" of the view.
  • 3D Submap Representation: The global 3DGS map is partitioned into spatial submaps. Features for these submaps are extracted using PointNet++, which processes the 3D coordinates and attributes of the Gaussians.
  • Feature Alignment: A Multi-Head Self-Attention mechanism, trained via contrastive learning, aligns the 2D semantic features with the 3D submap features. This allows the system to retrieve the most likely submap and estimate a coarse initial pose.

2.2 Fine Stage: Rendering-based Refinement

Once a coarse pose is established, SGLoc treats the 3DGS map as a differentiable environment to refine the camera's position and orientation.

  • Differentiable Rendering: The system renders a synthetic view from the current pose estimate using the 3DGS engine.
  • Optimization Loop: The camera pose is iteratively updated to minimize a composite loss function. This function includes a pixel-level photometric loss and a geometric matching loss derived from LoFTR (Local Feature Transformer) to ensure perfect alignment.
  • Quality Control: To ensure reliability, the system calculates the Peak Signal-to-Noise Ratio (PSNR) between the query and rendered images. If the PSNR falls below a specific threshold, the retrieved pose is flagged as a potential outlier and discarded, preventing the refinement stage from converging on a false positive.

3. Experimental Evaluation

The performance of SGLoc was validated on the 7Scenes and 12Scenes indoor datasets, which are characterized by challenging motion blur, reflections, and complex layouts.

3.1 Accuracy and Robustness

SGLoc demonstrated superior precision compared to existing state-of-the-art (SOTA) methods. On the 7Scenes dataset, the system achieved a median translation error of 0.15 cm and a rotation error of 0.05°. This represents a 29% improvement over previous 3DGS-based baselines. On the 12Scenes dataset, the system maintained high accuracy with a median error of 0.3 cm and 0.2°, showing up to an 87.5% improvement in translation accuracy over traditional regression-based approaches in specific scenes.

3.2 Global Localization Capability

A key strength of SGLoc is its ability to recover the camera pose without any initial guess. While many competing methods fail if the starting point is more than a few centimeters away, SGLoc successfully localizes images even when the initial discrepancy spans several meters or significant angular offsets. This makes it a true "global" localization solution.

4. Discussion and Limitations

While SGLoc sets a new benchmark for 3DGS-based localization, several areas for improvement remain. The current reliance on fixed spatial submaps may not scale optimally to extremely large or sparse outdoor environments. Additionally, the iterative nature of the refinement stage introduces computational overhead that may limit real-time deployment on low-power mobile platforms. Finally, the system is currently designed for static scenes; future work will explore the integration of dynamic object handling within the semantic framework.

5. Conclusion

SGLoc introduces a powerful paradigm for visual localization by combining the high-fidelity rendering of 3D Gaussian Splatting with robust semantic feature matching. By successfully addressing the global localization problem and achieving sub-centimeter accuracy, SGLoc provides a reliable "semantic compass" for the next generation of autonomous systems.

Summary updated

BibTeX
@inproceedings{xu2025sgloc,
  author = {Beining Xu and Siting Zhu and Hesheng Wang},
  title = {{SGLoc: Semantic Localization System for Camera Pose Estimation from 3D Gaussian Splatting Representation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11245850},
}
Figure