Neural Reconstruction from Unconstrained In-the-Wild Image Collections

This direction is merged into another direction: photometric inconsistency. Its last version stays readable and citable here.

Version 1 of 188 papers92 conceptsfrozen 2026-10-04

Most 3D reconstruction algorithms operate under a "Goldilocks" assumption: they require a dense set of images where the lighting is perfectly static, the camera settings are fixed, and nothing moves. In the real world—specifically when using "in-the-wild" data like tourist photos from Flickr or large-scale urban captures—this assumption of photometric consistency collapses.

A single physical point on a building might appear bright yellow in a sunset photo, grey in a rainy-day shot, and nearly black in a nighttime capture. Standard models like NeRF or 3D Gaussian Splatting (3DGS) interpret these color discrepancies as physical density, resulting in "foggy" geometry or floating artifacts as the model tries to average contradictory pixel data.

The research direction emerging from these papers focuses on decoupling the intrinsic scene properties from transient environmental factors, allowing for high-fidelity 3D reconstruction from chaotic, unconstrained image collections.

The Photometric Consistency Problem

Traditional Multi-View Stereo and early neural rendering assume that a point's radiance is a function only of its position and viewing angle. However, as noted by Georg Kuschk and Filip Radenovic, large-scale outdoor datasets are plagued by inconsistent illumination.

When a model encounters two images of the same pillar—one overexposed and one underexposed—it faces a mathematical conflict. If it treats the color as "truth," it must conclude that the pillar is semi-transparent or that there is "ghost" geometry in front of it to explain the brightness change. This is why VastGaussian and City-on-Web identify photometric discrepancies as a primary cause of visual artifacts in urban reconstruction.

Disentangling Appearance via Latent Embeddings

The breakthrough for handling these inconsistencies, pioneered by Ricardo Martin Brualla and Jonathan T. Barron in NeRF in the Wild, is to treat "appearance" as a separate, per-image variable.

Instead of forcing the network to learn a single color for a point, researchers provide the network with an appearance embedding—a low-dimensional vector that describes the specific lighting and weather of a single photo. This allows the model to learn a shared "base" geometry while delegating the "accidental" lighting effects to the embedding. Ha-NeRF and AAGS extend this by hallucinating missing information or using appearance-aware splatting to ensure that the static geometry remains sharp even when the lighting shifts drastically.

Intrinsic Decomposition: Albedo vs. Shading

A more physically grounded approach involves decomposing the scene into albedo (the inherent color of the material) and shading (the effect of light hitting that material).

In ExtremeNeRF, SeokYeong Lee and colleagues enforce "cross-view albedo consistency." The intuition is simple: while the shadows on a statue change as the sun moves, the color of the stone itself does not. By forcing the model to find a single albedo map that explains all images when combined with different shading layers, the reconstruction becomes robust to "unconstrained illumination." Gaussian in the Wild applies a similar logic to 3DGS, decoupling intrinsic material properties from dynamic environmental factors to achieve 1000x faster rendering than NeRF-based counterparts.

Modeling the Camera Pipeline

Sometimes the inconsistency isn't in the world, but in the camera. Kim Jun-Seong and Tae-Hyun Oh argue in HDR-Plenoxels that we must model the camera's internal imaging pipeline—including auto-exposure and white balance.

By "self-calibrating" these parameters, models like HDR-GS can reconstruct a High Dynamic Range (HDR) radiance field from a set of Low Dynamic Range (LDR) photos with mismatched exposures. This effectively "undoes" the camera's processing to find the true radiance of the scene, as explored by Xin Huang and Qi Zhang in their work on implicit camera models.

Filtering Transient Occluders

"In-the-wild" photos aren't just inconsistently lit; they are often cluttered with tourists, cars, and birds. These are transient occluders—objects that appear in one frame but not the next.

To solve this, researchers like Jingjie Lin in HybridGS and Myeongseok Nam in ForestSplats use dual-representation systems. They use a stable 3D representation (like 3D Gaussians) for the static background and a per-image 2D "transient field" to "catch" the moving objects. By giving the optimization a place to "put" the tourists (the transient field), the model no longer tries to bake them into the permanent 3D geometry of the landmark.

From Sparse Views to Generative Priors

The most recent frontier, represented by SparseGS-W and MS-GS, tackles the "extreme" version of this problem: reconstructing a scene from only a handful (e.g., five) unconstrained images.

When data is this sparse, disentanglement isn't enough; the model needs a "prior" of what the world looks like. By using multiview generative diffusion models, researchers like Philipp Henzler can "relight" all input images to a single reference illumination before the reconstruction even begins. This effectively synthesizes the "Goldilocks" conditions that the reconstruction algorithms originally required, bridging the gap between casual photography and professional 3D capture.

Papers in this version

Showing 60 of 88 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_neural_reconstruction_from_unconstrained_87092158,
  title = {Neural Reconstruction from Unconstrained In-the-Wild Image Collections},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-reconstruction-from-unconstrained-in-the-wild-image-collections/rdr_8709215806334228981c56d7bd70a327},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_neural_reconstruction_from_unconstrained_87092158_v1,
  title = {Neural Reconstruction from Unconstrained In-the-Wild Image Collections},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-reconstruction-from-unconstrained-in-the-wild-image-collections/rdr_8709215806334228981c56d7bd70a327/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}