Bypassing Per Scene Optimization in 3D Scene Reconstruction

521 concepts378 papers7 proposals

Early breakthroughs in neural rendering, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), achieved photorealism by treating neural networks or explicit primitives as scene-specific lookup tables. To reconstruct a scene, a model had to be optimized from scratch using thousands of iterations of gradient descent on a dedicated set of images.

As Bao et al. note in their comprehensive survey, 3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities, this reliance on scene-specific optimization creates a massive bottleneck. It prevents models from generalizing to new environments without lengthy retraining, making immediate deployment in dynamic or unseen settings impossible.

To bridge this zero-shot generalization gap, a prominent research direction focuses on generalizable feed-forward neural rendering. Instead of memorizing a single scene through test-time optimization, these methods use feed-forward networks to directly predict 3D structures from sparse input images in a single forward pass.

The Memorization Bottleneck

When a neural rendering model requires per-scene optimization, it acts less like an intelligent visual system and more like a lossy compression algorithm for a specific set of photos. For instance, Thies et al. point out in Image-guided Neural Object Rendering that this introduces a significant time delay between data acquisition and the ability to render novel views, limiting interactive applications. Similarly, Häni et al. emphasize in Continuous Object Representation Networks: Novel View Synthesis without Target View Supervision that continuous scene representations lack near-instant inference capabilities because they must be optimized for each specific object at test time.

This bottleneck restricts 3D vision from scaling to real-time applications like robotics, augmented reality, or rapid medical imaging. If a system cannot generalize across scenes out-of-the-box, it cannot navigate an unknown room, track a moving subject, or visualize a new medical scan instantly.

From Optimization to Inference

Generalizable models replace test-time gradient descent with feed-forward inference. The network takes a few source images and maps their features directly into a 3D representation.

Early foundations for this approach showed that networks could learn cross-scene priors to synthesize views immediately after a quick geometric scan, as demonstrated by Riegler et al. in Free View Synthesis. More recently, this paradigm has been adapted to explicit representations. For example, Chen et al. introduced FreeSplat: Generalizable 3D Gaussian Splatting Towards Free View Synthesis of Indoor Scenes, which bypasses per-scene optimization by using efficient CNN-based cost volumes and pixel-wise fusion modules to predict 3D Gaussian primitives directly from indoor image sequences.

Mechanisms of Generalization

To eliminate per-scene training, generalizable architectures rely on three main strategies to map 2D pixels to 3D space:

  1. Direct Parameter Regression: Instead of optimizing the positions, opacities, and colors of 3D primitives via gradient descent, feed-forward networks regress these parameters directly from pixel features. In GPS-Gaussian: Generalizable Pixel-Wise 3D Gaussian Splatting for Real-Time Human Novel View Synthesis, Zheng et al. regressed 3D Gaussian parameters directly from sparse source images, enabling real-time human novel view synthesis without any subject-specific optimization.
  2. Geometry-Aware Feature Mapping: Networks can project 2D image features into a shared 3D space (such as voxels or multiplane images) using epipolar geometry. Li et al. combined Multiplane Images (MPI) with continuous representations in MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis to achieve fast depth reconstruction across unseen scenes. Similarly, Zhao developed MVPSNet: Fast Generalizable Multi-view Photometric Stereo using "Light Aggregated Feature Maps" to generalize across unseen objects, reducing processing times from 12 hours to under two minutes.
  3. Incorporating Structural Priors: When operating within specific domains, models can leverage structural templates to guide feed-forward prediction. Dey proposed GHNeRF: Learning Generalizable Human Features with Efficient Neural Radiance Fields to integrate a human-feature encoder into a neural radiance field pipeline, allowing the model to generalize to unseen human figures instantly. For dynamic interactions, Liu et al. developed Generalizable Hand-Object Modeling from Monocular RGB Images via 3D Gaussians, using a vision-driven perception module to achieve cross-scene generalization for complex hand-object interactions in seconds.

Why It Works: Learning the Mapping, Not the Scene

The core intuition behind generalizable neural rendering lies in a fundamental shift in what the network weights represent.

In a scene-specific model, the network weights learn a direct mapping from spatial coordinates to color and density ($x, y, z \to \text{color, density}$). If the scene geometry changes, those coordinates lose their meaning. This is why Jeong notes in 3D Scene Painting via Semantic Image Synthesis that coordinate-based synthesis frameworks require a dedicated training process for each unique layout.

In contrast, generalizable models learn a conditional mapping ($\text{Images} \times \text{Geometry} \to \text{3D Representation}$). During a large-scale pre-training phase across diverse environments, the network learns fundamental principles of computer vision: how perspective projection works, how light reflects off surfaces, and how to resolve occlusions by comparing multiple viewpoints.

Once these geometric and appearance priors are baked into the network weights, the model no longer needs to optimize from scratch. It can look at a handful of images from a completely novel scene, use its learned priors to infer the underlying geometry, and output the 3D structure instantly. This enables downstream tasks like 3D object detection (Xu et al.), unified image restoration (Gupta et al.), and autonomous driving simulation (Han et al.) to operate in real-time without the computational overhead of per-scene optimization.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 412 papers
  2. Version 1 · 2026-10-04 · 291 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_per_scene_optimization_bottleneck_dba624d3,
  title = {per-scene optimization bottleneck},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/per-scene-optimization-bottleneck/rdr_dba624d30f174422a018cbc0a94e0292},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_per_scene_optimization_bottleneck_dba624d3_v2,
  title = {per-scene optimization bottleneck},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/per-scene-optimization-bottleneck/rdr_dba624d30f174422a018cbc0a94e0292/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake