GGRt: Towards Pose-Free Generalizable 3D Gaussian Splatting in Real-Time

Hao Li, Yuanyuan Gao, Chenming Wu, Dingwen Zhang, Yalun Dai, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, Junwei Han

Introduction

State-of-the-art 3D reconstruction and novel view synthesis methods, such as standard 3D Gaussian Splatting (3D-GS) and Neural Radiance Fields (NeRF), typically rely on a "pose-first" workflow. These methods require accurate camera poses, often obtained via computationally expensive Structure-from-Motion (SfM) algorithms like COLMAP. This dependency limits their utility in real-time applications or scenarios where camera metadata is unavailable. Furthermore, generalizable models—designed to work on unseen scenes without per-scene optimization—often face high memory requirements for high-resolution images and slow rendering speeds due to repeated network forward passes.

GGRt addresses these challenges by introducing the first "pose-free" generalizable 3D-GS framework. It jointly learns pose estimation and 3D representation, achieving reconstruction speeds of over 5 FPS and rendering speeds exceeding 100 FPS.

Methodology

The GGRt framework integrates two primary components for joint learning:

  1. Iterative Pose Optimization Network (IPO-Net): Drawing inspiration from the RAFT architecture for optical flow, IPO-Net estimates relative camera poses and depth maps directly from image observations. It utilizes a Conv-GRU module to iteratively minimize feature-metric reprojection errors across views.
  2. Generalizable 3D-Gaussians (G-3DG): This module predicts 3D Gaussian primitives from image pairs. It employs an Epipolar Sampler for correspondence search, Cross-Attention for feature aggregation, and a Local Self-Attention module to maintain spatial consistency.
The GGRt pipeline facilitates joint learning of poses and Gaussians, utilizing a caching mechanism for efficiency.

To enhance practicality, GGRt incorporates two key innovations:

  • Deferred Back-propagation (DBP): To overcome GPU memory limits during high-resolution training, GGRt first renders a full-resolution image without tracking gradients. It then re-renders specific patches with gradients enabled to update the network, allowing the model to learn from high-resolution details.
  • Gaussians Cache Mechanism: For video sequences, reference views are often reused. GGRt stores predicted Gaussians in a dynamic cache, significantly accelerating inference by avoiding redundant computations.

Experimental Evaluation

GGRt was evaluated on the LLFF, Waymo Open, and KITTI datasets. The model demonstrates robust generalization to new sensor setups without fine-tuning.

Key Results

GGRt establishes a new state-of-the-art for pose-free novel view synthesis:

  • Real-time Performance: The system achieves $\ge$ 5 FPS for full reconstruction and $\ge$ 100 FPS for rendering.
  • High Fidelity: On the Waymo dataset, GGRt achieved a PSNR of approximately 32 dB, significantly outperforming other pose-free methods like DBARF (~24 dB).
  • Efficiency: The Gaussians Cache mechanism provides an 8x speedup during inference compared to a non-cached baseline.
Qualitative comparison on the Waymo dataset demonstrates GGRt's superior reconstruction quality and real-time metrics compared to existing methods.

Discussion and Conclusion

By transitioning from implicit representations (NeRF) to explicit ones (3D-GS), GGRt decouples scene generation from rendering. Once Gaussians are cached, rendering proceeds via standard rasterization, enabling high speeds. Joint training of IPO-Net and G-3DG ensures that estimated poses are optimized specifically for 3D reconstruction.

While GGRt represents a significant advancement, it still requires visual overlap between reference views for accurate geometry estimation. Future work may explore more complex outdoor environments and global loop closure to mitigate pose drift in long sequences.

Summary updated

Abstract

This paper presents GGRt, a novel approach to generalizable novel view synthesis that alleviates the need for real camera poses, complexity in processing high-resolution images, and lengthy optimization processes, thus facilitating stronger applicability of 3D Gaussian Splatting (3D-GS) in real-world scenarios. Specifically, we design a novel joint learning framework that consists of an Iterative Pose Optimization Network (IPO-Net) and a Generalizable 3D-Gaussians (G-3DG) model. With the joint learning mechanism, the proposed framework can inherently estimate robust relative pose information from the image observations and thus primarily alleviate the requirement of real camera poses. Moreover, we implement a deferred back-propagation mechanism that enables high-resolution training and inference, overcoming the resolution constraints of previous methods. To enhance the speed and efficiency, we further introduce a progressive Gaussian cache module that dynamically adjusts during training and inference. As the first pose-free generalizable 3D-GS framework, GGRt achieves inference at \(\ge \)5 FPS and real-time rendering at \(\ge \)100 FPS. Through extensive experimentation, we demonstrate that our method outperforms existing NeRF-based pose-free techniques in terms of inference speed and effectiveness. It can also approach the real pose-based 3D-GS methods. Our contributions provide a significant leap forward for the integration of computer vision and computer graphics into practical applications, offering state-of-the-art results on LLFF, KITTI, and Waymo Open datasets and enabling real-time rendering for immersive experiences. Project page: https://3d-aigc.github.io/GGRt.
BibTeX
@inproceedings{li2024ggrt,
  author = {Hao Li and Yuanyuan Gao and Chenming Wu and Dingwen Zhang and Yalun Dai and Chen Zhao and Haocheng Feng and Errui Ding and Jingdong Wang and Junwei Han},
  title = {{GGRt: Towards Pose-Free Generalizable 3D Gaussian Splatting in Real-Time}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=4chhv5h58z},
}

Loading related content...

Figure