Joint Optimization of Camera Pose and Neural 3D Geometry

This direction is split into other directions: joint pose-geometry optimization (interdependent parameter estimation). Its last version stays readable and citable here.

Version 1 of 1125 papers125 conceptsfrozen 2026-10-04

To reconstruct a 3D scene from a set of images, you traditionally need to know exactly where the camera was for every shot. This is the "pose" of the camera. In standard pipelines, researchers use tools like COLMAP to solve for these poses using Structure-from-Motion (SfM) before they even begin training a Neural Radiance Field (NeRF) or 3D Gaussian Splatting (3DGS) model.

However, as noted by Onur Özyeşil and colleagues in their Survey on Structure from Motion, this creates a fundamental interdependency: you need precise camera poses to determine 3D geometry, but you often need an understanding of the 3D structure to estimate those poses accurately. This research direction focuses on "pose-free" or "unconstrained" reconstruction, where the system learns the camera positions and the 3D scene simultaneously.

The Circular Dependency Problem

The core challenge in this field is a "chicken-and-egg" or circular dependency. If you start with incorrect camera poses, the neural model (like a NeRF) will try to minimize the reconstruction error by "baking" those errors into the geometry. For example, if a camera is slightly to the left of where the model thinks it is, the model might create a "ghosting" artifact or a distorted surface to make the pixels align.

Once the geometry is distorted, it becomes impossible to refine the camera pose further because the "anchor"—the 3D scene—is no longer physically plausible. As Injae Kim explains in UP-NeRF, this joint optimization often leads to failed convergence or local minima where the model finds a mathematically consistent but physically impossible solution.

Strategy 1: Differentiable Bundle Adjustment

One way to break this cycle is to treat camera poses as learnable parameters that are optimized via backpropagation alongside the neural field. This is often called "Bundle-Adjusting" the neural representation.

In L2G-NeRF, Yue Chen and others propose a local-to-global registration process. Instead of trying to solve the whole scene at once, the model performs local alignment that gradually expands. This prevents the optimization from getting trapped in the "messy" local minima that occur when large initial pose misalignments cause the neural field to collapse. Similarly, KeyGS by Keng Wei Chang applies this to Gaussian Splatting, using a coarse-to-fine strategy to ensure that the geometry and poses refine each other in a stable, synchronized way.

Strategy 2: Alternating Optimization and Priors

If optimizing both at once is too unstable, why not take turns? Kun Wang and the AltNeRF team introduced a framework that alternates between self-supervised monocular depth estimation and NeRF optimization.

The intuition here is to use a "prior"—a rough guess of what the 3D world looks like—to guide the pose estimation. By using a depth-estimation network to provide a geometric starting point, the pose optimization has a "map" to follow. As the poses improve, they provide better multi-view supervision to the depth network, creating a "virtuous cycle" of refinement. SAMURAI, developed by Andreas Engelhardt and Jonathan Barron, extends this to "in-the-wild" images, jointly solving for shape, material, and illumination alongside pose to handle the inconsistent lighting of internet photo collections.

Strategy 3: Generative Lifting and Foundation Models

The most recent frontier involves using "foundation models" to bypass the need for an initial pose estimate entirely. Instead of starting from a random guess, researchers are "lifting" 2D images into 3D using the spatial intelligence of pre-trained models.

Rana Hanocka and colleagues demonstrated this in Generative Lifting of Multiview to 3D by wrapping a NeRF inside a Diffusion model. The diffusion process acts as a powerful regularizer; it "knows" what a realistic object looks like from different angles. This allows the system to denoise the 3D representation and the camera poses simultaneously. ZeroGS by Yu Chen follows a similar logic, using 3D foundation models to provide a progressive registration pipeline that can organize hundreds of unordered, unposed images into a coherent 3D Gaussian Splatting model.

Why It Works: The Geometric Anchor

The reason these methods work—where simple joint optimization fails—is that they all find a way to provide a geometric anchor.

  • In Bundle-Adjusting methods, the anchor is a coarse-to-fine frequency schedule that prevents high-frequency noise from ruining the pose gradients.
  • In Alternating methods, the anchor is a depth or silhouette prior that constrains the possible shapes.
  • In Generative methods, the anchor is a pre-trained understanding of 3D space.

By providing this anchor, researchers resolve the circular dependency, allowing the camera poses to "find" the geometry and the geometry to "lock in" the poses. This moves us closer to a world where we can reconstruct high-fidelity 3D scenes from any collection of photos, regardless of whether we know where the photographer was standing.

Papers in this version

Showing 60 of 125 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_joint_optimization_of_camera_pose_and_ne_f403f2e9,
  title = {Joint Optimization of Camera Pose and Neural 3D Geometry},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-optimization-of-camera-pose-and-neural-3d-geometry/rdr_f403f2e991f1435ea76909aa86496bcd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_joint_optimization_of_camera_pose_and_ne_f403f2e9_v1,
  title = {Joint Optimization of Camera Pose and Neural 3D Geometry},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-optimization-of-camera-pose-and-neural-3d-geometry/rdr_f403f2e991f1435ea76909aa86496bcd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}