Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization

Christian Schmidt, Jens Piekenbrinck, Bastian Leibe

Abstract

Traditional 3D Gaussian Splatting (3DGS) relies on accurate camera poses, typically obtained via computationally intensive Structure from Motion (SfM) pipelines. We introduce a method to perform high-quality 3D reconstruction and novel view synthesis (NVS) without requiring precise initial poses. By integrating differentiable camera pose optimization directly into 3DGS CUDA kernels and introducing a novel anisotropy loss, our approach achieves state-of-the-art results with significant speedups—up to 40x faster than previous NeRF-based pose-free methods.

Introduction

Modern 3D reconstruction techniques like 3D Gaussian Splatting have revolutionized novel view synthesis. However, their dependence on "posed" images—where camera extrinsics are known—remains a significant bottleneck. Standard SfM tools like COLMAP are slow and often fail in challenging environments with repetitive patterns or low texture. While "pose-free" methods like BARF or Nope-NeRF address this, they are built on Neural Radiance Fields (NeRFs), which are notoriously slow to train and render. This paper bridges the gap by enabling pose-free capabilities within the high-performance 3DGS framework.

Methodology

The proposed framework extends standard 3DGS through three primary innovations:

  1. Differentiable Pose Optimization: Camera extrinsics (rotation and translation) are treated as learnable parameters within the SE(3) Lie Group. Analytical gradients are derived and integrated into the high-performance CUDA rasterizer, allowing the system to refine camera positions based on photometric error with minimal overhead.
  2. Anisotropy Loss ($L_{An}$): To prevent the model from "cheating" by stretching Gaussians into thin shards to fit incorrect poses, this loss term penalizes excessive eccentricity. This forces the optimization to correct the camera poses rather than distorting the scene geometry.
  3. Adaptive Thresholding and Pruning: The model prunes semi-transparent Gaussians to resolve shape-radiance ambiguity, ensuring the resulting geometry is coherent and opaque.

For video sequences with entirely unknown poses, a "warm-up" phase utilizes a monocular depth estimator (DPT) to initialize a rough point cloud, followed by iterative relative pose estimation between consecutive frames.

Key Results and Discussion

The method was evaluated on standard benchmarks including LLFF, Replica, and Tanks and Temples.

Performance and Accuracy

Our approach demonstrates remarkable efficiency. On the LLFF dataset, it achieves superior results in just 10 minutes, compared to 300 minutes for BARF—a 30x speedup. In terms of accuracy, it outperformed the previous state-of-the-art (Nope-NeRF) on the Tanks and Temples dataset, achieving a PSNR of 31.24 versus 26.34.

Convergence Robustness

The system is highly robust to initial pose noise. Even with rotation errors up to 15 degrees, the model successfully converges to the correct trajectory. The integration of pose gradients into the CUDA kernels makes this refinement step nearly "free" in terms of computational time.

Conclusion

"Look Gauss, No Pose" effectively brings the speed and quality of 3D Gaussian Splatting to the pose-free domain. By baking pose refinement into the core rendering kernels and regularizing geometry via anisotropy loss, the authors provide a robust alternative to slow NeRF-based optimization. While the method still requires reasonable image overlap, it represents a major step toward real-time, autonomous 3D reconstruction from unposed imagery.

Summary updated

BibTeX
@inproceedings{schmidt2024look,
  author = {Christian Schmidt and Jens Piekenbrinck and Bastian Leibe},
  title = {{Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801639},
}
Figure