Joint Optimization of Camera Poses and Neural Geometry

952 concepts557 papers15 proposals

Traditional 3D reconstruction pipelines typically follow a rigid two-stage process: first, estimate camera poses using Structure-from-Motion (SfM) tools like COLMAP; second, reconstruct the scene geometry using those fixed poses. However, this "chicken-and-egg" problem creates a bottleneck. If the initial pose estimation is slightly off due to sensor noise, motion blur, or lack of texture, the subsequent reconstruction is doomed to be blurry or topologically incorrect.

A growing body of research proposes a more fluid alternative: joint pose-geometry optimization. By treating camera parameters not as static inputs but as learnable variables within a differentiable rendering framework, these methods allow the scene's emerging 3D structure to "pull" the camera poses into alignment.

The Differentiable Camera

The core mechanism behind this shift is the integration of camera extrinsics (position and orientation) and intrinsics (focal length and lens distortion) directly into the optimization loop. Instead of being fixed, these parameters are updated via backpropagation alongside the neural weights of the scene representation.

To ensure stable updates, researchers often parameterize camera extrinsics within the SE(3) Lie Group. As Zhu explains, performing optimization on the tangent plane (the Lie algebra) rather than directly on the complex manifold allows for smoother convergence. This approach enables models like Multi-NeuS and CodeNeRF to recover accurate 3D structures and viewpoints simultaneously, even when starting from unposed images or noisy metadata.

Analysis-by-Synthesis as a Correction Signal

Why does this work? The intuition lies in the analysis-by-synthesis loop. When a model renders a virtual view from a guessed camera pose, it compares that rendering to the actual observed image. The resulting photometric error serves as a supervisory signal.

If the rendered pixels don't match the observed pixels, the error could stem from two sources: the 3D model is wrong, or the camera is looking from the wrong spot. Because the entire pipeline—from the camera ray generation to the final pixel color—is differentiable, the system can compute exactly how to "nudge" the camera to minimize the discrepancy. Schmidt et al. demonstrate this with 3D Gaussian Splatting, showing that analytical gradients from photometric error can refine camera positions up to 40x faster than previous NeRF-based methods. Similarly, Heng Li et al. show that this end-to-end alignment allows for dense SLAM using only monocular RGB input, as the internal consistency of the neural map acts as a geometric anchor for the trajectory.

Correcting Real-World Distortions

This joint optimization is particularly powerful for handling the imperfections of real-world sensors. Traditional geometric methods often struggle with "non-ideal" cameras, but differentiable refinement can model these artifacts explicitly:

  • Rolling Shutter and Motion Blur: When a camera moves quickly, images suffer from "jello" distortion or blur. Moyang Li et al. and Zhao et al. integrate continuous-time trajectories (like B-splines) into the optimization. By modeling the row-by-row formation of the image, they can simultaneously "undistort" the scene and recover sharp geometry from motion-blurred frames.
  • Lens Distortion: Bozic et al. introduced differentiable lens models using invertible neural networks. This allows a system to learn the specific warping of a fisheye or wide-angle lens at the same time it learns the scene's shape, preventing the "foggy" artifacts that occur when a model tries to fit a distorted image to a pinhole camera model.
  • Hardware Misalignment: For robots or multi-camera rigs, physical vibrations can ruin factory calibrations. Yang et al. and Tianjia Zhang et al. treat these extrinsics as learnable parameters, allowing the system to "self-calibrate" in the field by aligning multiple sensors to a shared neural radiance field.

Beyond Static Scenes

The principle of joint refinement extends to dynamic subjects, such as humans. In these cases, the "pose" includes not just the camera's location but the configuration of a skeletal model. Te et al. and Jiang et al. demonstrate that backpropagating rendering errors to a parametric body model (like SMPL) corrects for tracking drift. By ensuring the digital skeleton matches the image pixels more precisely, the system achieves significantly sharper reconstructions of moving actors than static pose estimation could provide.

Ultimately, this research direction suggests that geometry and pose are two sides of the same coin. As Gupta et al. and Jang et al. argue, neural optimization can often reach a higher level of accuracy than traditional SfM pipelines because it leverages the full photometric richness of the image rather than just a few sparse feature points.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 553 papers
  2. Version 1 · 2026-10-04 · 656 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_joint_pose_geometry_optimization_differe_3d6eee3c,
  title = {joint pose-geometry optimization (differentiable pose refinement)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-pose-geometry-optimization-differentiable-pose-refinement/rdr_3d6eee3cf6644f3eaec74cf67d976159},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_joint_pose_geometry_optimization_differe_3d6eee3c_v2,
  title = {joint pose-geometry optimization (differentiable pose refinement)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-pose-geometry-optimization-differentiable-pose-refinement/rdr_3d6eee3cf6644f3eaec74cf67d976159/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake