Introduction
State-of-the-art 3D reconstruction and novel view synthesis methods, such as standard 3D Gaussian Splatting (3D-GS) and Neural Radiance Fields (NeRF), typically rely on a "pose-first" workflow. These methods require accurate camera poses, often obtained via computationally expensive Structure-from-Motion (SfM) algorithms like COLMAP. This dependency limits their utility in real-time applications or scenarios where camera metadata is unavailable. Furthermore, generalizable models—designed to work on unseen scenes without per-scene optimization—often face high memory requirements for high-resolution images and slow rendering speeds due to repeated network forward passes.
GGRt addresses these challenges by introducing the first "pose-free" generalizable 3D-GS framework. It jointly learns pose estimation and 3D representation, achieving reconstruction speeds of over 5 FPS and rendering speeds exceeding 100 FPS.
Methodology
The GGRt framework integrates two primary components for joint learning:
- Iterative Pose Optimization Network (IPO-Net): Drawing inspiration from the RAFT architecture for optical flow, IPO-Net estimates relative camera poses and depth maps directly from image observations. It utilizes a Conv-GRU module to iteratively minimize feature-metric reprojection errors across views.
- Generalizable 3D-Gaussians (G-3DG): This module predicts 3D Gaussian primitives from image pairs. It employs an Epipolar Sampler for correspondence search, Cross-Attention for feature aggregation, and a Local Self-Attention module to maintain spatial consistency.
To enhance practicality, GGRt incorporates two key innovations:
- Deferred Back-propagation (DBP): To overcome GPU memory limits during high-resolution training, GGRt first renders a full-resolution image without tracking gradients. It then re-renders specific patches with gradients enabled to update the network, allowing the model to learn from high-resolution details.
- Gaussians Cache Mechanism: For video sequences, reference views are often reused. GGRt stores predicted Gaussians in a dynamic cache, significantly accelerating inference by avoiding redundant computations.
Experimental Evaluation
GGRt was evaluated on the LLFF, Waymo Open, and KITTI datasets. The model demonstrates robust generalization to new sensor setups without fine-tuning.
Key Results
GGRt establishes a new state-of-the-art for pose-free novel view synthesis:
- Real-time Performance: The system achieves $\ge$ 5 FPS for full reconstruction and $\ge$ 100 FPS for rendering.
- High Fidelity: On the Waymo dataset, GGRt achieved a PSNR of approximately 32 dB, significantly outperforming other pose-free methods like DBARF (~24 dB).
- Efficiency: The Gaussians Cache mechanism provides an 8x speedup during inference compared to a non-cached baseline.
Discussion and Conclusion
By transitioning from implicit representations (NeRF) to explicit ones (3D-GS), GGRt decouples scene generation from rendering. Once Gaussians are cached, rendering proceeds via standard rasterization, enabling high speeds. Joint training of IPO-Net and G-3DG ensures that estimated poses are optimized specifically for 3D reconstruction.
While GGRt represents a significant advancement, it still requires visual overlap between reference views for accurate geometry estimation. Future work may explore more complex outdoor environments and global loop closure to mitigate pose drift in long sequences.
Summary updated