Generative Priors for Extrapolated Novel View Synthesis

This direction is merged into another direction: viewpoint extrapolation limits. Its last version stays readable and citable here.

Version 1 of 146 papers46 conceptsfrozen 2026-10-04

When you train a 3D reconstruction model like 3D Gaussian Splatting (3DGS) or a Neural Radiance Field (NeRF), you are essentially teaching a computer to memorize a scene from specific angles. If you move the virtual camera slightly between those training frames, the results are stunning. This is interpolation, and we have largely solved it.

However, as soon as you "step off the path"—for example, by moving a virtual car into a lane it never actually drove in—the scene typically dissolves. You encounter what Zeyu Yang and colleagues call the "tunnel vision problem" in Driving Scene Synthesis on Free-form Trajectories with Generative Prior. The model lacks the geometric and textural information to describe what was behind a corner or under a car. This is the challenge of extrapolation, and a new wave of research is solving it by marrying 3D reconstruction with the "imagination" of generative AI.

The Extrapolation Gap

Traditional 3D reconstruction is grounded in evidence. If a pixel wasn't seen by the camera, the model doesn't know what color it should be. In VEGS: View Extrapolation of Urban Scenes in 3D Gaussian Splatting Using Learned Priors, Sungwon Hwang and Jayeon Kang describe how this leads to "floaters," broken textures, and gaping geometric cavities when the camera orientation deviates from the training data.

The core insight of recent work is that while 3D models provide the structure, generative models (like Diffusion models or GANs) provide the prior knowledge. A 3DGS model might not know what the sidewalk looks like from a top-down view, but a diffusion model trained on millions of images knows what any sidewalk generally looks like.

Bridging Structure and Imagination

Researchers are using two primary strategies to bridge this gap: Generative Refinement and Knowledge Distillation.

Generative Refinement: The "Fixer" Model

In this approach, a 3D model produces a "coarse" render that contains artifacts or holes. A generative model then "fixes" the image. Kunhao Liu and Shijian Lu demonstrate this in Novel View Extrapolation with Video Diffusion Priors, where they use Stable Video Diffusion to transform artifact-prone renders into high-fidelity video without fine-tuning the underlying 3D representation. Similarly, EVPGS by Jiahe Li uses a coarse-to-fine strategy to "fill in the blanks" for geometry that was never captured.

Knowledge Distillation: Teaching the 3D Model

Instead of fixing the output, some researchers bake generative knowledge directly into the 3D representation. DriveX progressively distills knowledge from a video diffusion model into a 3D Gaussian Splatting scene. This allows the 3D model itself to become "aware" of unseen regions, enabling real-time rendering of off-path trajectories like lane changes or turns.

Solving the "Off-Rails" Problem in Driving

The most critical application for this research is autonomous driving simulation. To train safe AI drivers, we need to simulate "what-if" scenarios: What if the car had turned left instead of going straight?

Standard models fail here because they are "on-rails"—restricted to the narrow corridor of the original sensor data. Lue Fan and Hongsheng Li address this with FreeSim, which enables high-fidelity rendering even meters away from the original path. By conditioning diffusion models on sparse 3D data—like the "pseudo-images" used by Qitai Wang in FreeVS—these systems maintain 3D geometric consistency while hallucinating photorealistic details for the new perspective.

Beyond the Training Path: Unbounded Exploration

The ultimate goal of this direction is "perpetual" or "unbounded" synthesis. Tianjiao Ding and Rene Vidal push this to the extreme with DynamicVoyager, which treats dynamic scene generation as a 3D-aware "outpainting" task. Instead of just interpolating between frames, the model can "voyage" into a scene, generating continuous camera movements that far exceed the boundaries of the original input.

By combining the rigorous spatial constraints of 3D geometry with the creative capacity of generative priors, this research is moving us from static 3D "reconstructions" toward truly explorable, hallucinated worlds. As Haoru Wang suggests in UP-LVSM, the future of the field may lie in stripping away handcrafted 3D biases and allowing models to learn the underlying relationship between views directly from unposed data.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generative_priors_for_extrapolated_novel_5045d1de,
  title = {Generative Priors for Extrapolated Novel View Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-priors-for-extrapolated-novel-view-synthesis/rdr_5045d1de281449beaa782d1a001f25b2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_generative_priors_for_extrapolated_novel_5045d1de_v1,
  title = {Generative Priors for Extrapolated Novel View Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-priors-for-extrapolated-novel-view-synthesis/rdr_5045d1de281449beaa782d1a001f25b2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}