Standard pipelines for 3D reconstruction, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), typically operate in two disconnected stages. First, an external Structure-from-Motion (SfM) tool like COLMAP estimates camera poses by matching features across images. Second, these fixed poses are used to train the neural representation.
This dependency creates a "non-differentiable bottleneck." As noted by Wenjing Bian and colleagues in NoPe-NeRF, SfM often fails in environments with low texture, repetitive patterns, or significant motion blur. When the first stage fails, the second stage becomes impossible. A growing body of research seeks to collapse these two stages into a single, end-to-end optimization process where camera poses and scene geometry are learned simultaneously.
The Challenge of Joint Optimization
The intuition behind "pose-free" rendering is simple: if we treat camera parameters as learnable variables, we can use the gradient of the photometric loss (the difference between a rendered pixel and a real pixel) to update both the scene's density and the camera's location.
However, this is a highly non-convex problem. If the initial camera poses are too far from the truth, the model gets stuck in local minima, resulting in "floaters" or a collapsed geometry. To solve this, researchers have introduced several mechanisms to guide the optimization.
Anchoring the Optimization with Priors
To prevent the model from drifting, many methods incorporate external geometric cues that do not rely on perfect feature matching.
- Monocular Depth and Flow: NoPe-NeRF uses monocular depth priors to constrain the geometry, while Xiaoming Zheng and Dan Xu in Flow-NeRF treat dense optical flow as a primary optimization goal. By ensuring that the movement of pixels between frames matches the underlying 3D motion, the system can recover poses even in textureless regions.
- Global Feature Tracks: Instead of relying on point-to-point matches, TrackGS by Dongbo Shi and others uses global feature tracks as "geometric anchors." This allows the model to maintain consistency across wide baselines and rapid camera movements.
- Implicit Regularization: IR-NeRF, developed by Fangneng Zhan and colleagues, introduces a "scene codebook" to learn the distribution of valid poses, bridging the gap between real and rendered images without explicit geometric priors.
Incremental Growth and Progressive Learning
Another strategy to avoid local minima is to build the scene piece-by-piece. Rather than optimizing all images at once, methods like CF-NeRF by Qiang Wang adopt an incremental learning strategy. By starting with a few frames and "growing" the trajectory, the model establishes a stable geometric foundation before attempting to integrate complex rotations.
This concept was extended to 3D Gaussian Splatting by Sifei Liu and Xiaolong Wang in COLMAP-Free 3D Gaussian Splatting. Because Gaussians are explicit geometric primitives (unlike the implicit volumes of NeRF), they can be added sequentially to the scene. This "progressive growing" reduced training times from 30 hours to just 2, demonstrating that explicit representations might be more naturally suited for pose-free optimization.
Handling Dynamics and Degradation
The ultimate goal of this research is to handle "in-the-wild" data where traditional SfM fails completely.
- Dynamic Scenes: Traditional SfM assumes the world is static. RoDynRF by Yu-Lun Liu and wildNeRF by Shuja Khalid address this by jointly optimizing poses while decoupling static backgrounds from moving objects.
- Low-Quality Input: When images are blurry or noisy, feature matching fails. RustNeRF uses 3D-aware restoration to handle "dirty" data, while E3NeRF by Jia Li leverages event cameras to provide high-speed temporal data, allowing for sharp reconstructions from severely motion-blurred video.
By removing the reliance on pre-computed poses, these methods move neural rendering toward a truly end-to-end workflow, capable of reconstructing the world from the same messy, casual videos that humans capture every day.