3D Reconstruction from Uncalibrated Multi-View Video

This direction is merged into another direction: camera calibration dependency. Its last version stays readable and citable here.

Version 1 of 142 papers45 conceptsfrozen 2026-10-04

Traditional 3D reconstruction is a "lab-bound" technology. To turn 2D images into a 3D model, algorithms typically require precise camera calibration—knowing exactly where every camera is (extrinsics) and the specific properties of its lens (intrinsics). This requirement makes high-fidelity 3D capture impossible for "in-the-wild" scenarios like handheld smartphone videos, internet footage, or ad-hoc drone swarms.

A growing body of research is moving toward uncalibrated multi-view reconstruction, developing methods that can infer 3D structure, human pose, and dynamic motion from uncoordinated video streams without pre-measured camera data.

The Geometry of Invariants

The most direct way to bypass camera calibration is to focus on properties that do not change regardless of the camera's position. Brian Gordon and Daniel Cohen-Or introduced this concept with FLEX (FLEX: Extrinsic Parameters-free Multi-view 3D Human Motion Reconstruction).

Instead of trying to triangulate 2D points into a global 3D coordinate system—which requires knowing the distance between cameras—FLEX learns to predict camera-invariant properties like 3D joint angles and bone lengths. Because a person's femur length remains constant whether viewed from the front or the side, the model can aggregate information across views to produce a smooth, accurate 3D skeleton without ever calculating where the cameras are located.

Humans as Calibration Objects

If the cameras aren't calibrated, can the subjects themselves provide the missing geometric information? In many "in-the-wild" scenes, the presence of a known structure—like a human body—acts as a "living" calibration target.

Jianping Yin and Alan Yuille demonstrated this with EasyRet3D (EasyRet3D: Uncalibrated Multi-View Multi-Human 3D Reconstruction and Tracking). By treating the humans in the scene as the calibration objects, the system can solve for camera parameters and 3D poses simultaneously. Similarly, Boyuan Jiang proposed Probabilistic Triangulation (Probabilistic Triangulation for Uncalibrated Multi-View 3D Human Pose Estimation), which treats camera poses as probability distributions. By iteratively refining these distributions based on the 2D observations of human joints, the system "discovers" the camera setup through the movement of the people within it.

From Optimization to Feed-Forward Transformers

Historically, uncalibrated reconstruction was a slow, iterative process. Early methods, such as those by Satoshi Yaguchi and Hideo Saito 0001, relied on deforming 3D meshes to match 2D silhouettes over many iterations (Improving Quality of Free-Viewpoint Image by Mesh Based 3D Shape Deformation).

Recent work has shifted this burden to transformer-based architectures that can handle uncalibrated data in a single "feed-forward" pass.

Weighting Reliability in Ad-Hoc Arrays

When cameras are uncalibrated and uncoordinated—such as a group of people filming the same event on their phones—some views are inevitably more reliable than others. Yitao Zhu and Sheng Wang 0014 addressed this with the Mixture of Uncalibrated Cameras (MUC) (MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction).

Instead of forcing all cameras into a single, potentially erroneous global coordinate system, MUC generates independent 3D predictions for each view. It then uses a neural network to "weight" these predictions based on factors like joint distance and surface normals. This allows the system to ignore occluded or blurry views dynamically, producing a high-fidelity 3D mesh that is more robust than any single-view or simple averaged result.

Scaling to the Real World

The ultimate goal of this research is to enable 3D capture at any scale. Recent advancements have pushed these uncalibrated techniques into massive outdoor environments. For instance, new frameworks for multi-camera rigs (On-the-fly Large-scale 3D Reconstruction from Multi-Camera Rigs) use hierarchical alignment to reconstruct kilometer-scale scenes from drones or helmets in minutes, proving that the transition from calibrated labs to the unconstrained world is not just possible, but increasingly efficient.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_3d_reconstruction_from_uncalibrated_mult_64e3eb09,
  title = {3D Reconstruction from Uncalibrated Multi-View Video},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/3d-reconstruction-from-uncalibrated-multi-view-video/rdr_64e3eb09ed9a4999ab49e8161b4231dd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_3d_reconstruction_from_uncalibrated_mult_64e3eb09_v1,
  title = {3D Reconstruction from Uncalibrated Multi-View Video},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/3d-reconstruction-from-uncalibrated-multi-view-video/rdr_64e3eb09ed9a4999ab49e8161b4231dd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}