Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines

Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, Abhishek Kar

Abstract

We present a practical and robust deep learning solution for capturing and rendering novel views of complex real world scenes for virtual exploration. Previous approaches either require intractably dense view sampling or provide little to no guidance for how users should sample views of a scene to reliably render high-quality novel views. Instead, we propose an algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields. We extend traditional plenoptic sampling theory to derive a bound that specifies precisely how densely users should sample views of a given scene when using our algorithm. In practice, we apply this bound to capture and render views of real world scenes that achieve the perceptual quality of Nyquist rate view sampling while using up to 4000x fewer views. We demonstrate our approach's practicality with an augmented reality smartphone app that guides users to capture input images of a scene and viewers that enable realtime virtual exploration on desktop and mobile platforms.

Introduction

Creating new views of a real-world scene from a set of photos—view synthesis—traditionally faces a trade-off between sampling density and geometric complexity. Light field rendering typically requires an astronomical number of photos (Nyquist sampling) to avoid "ghosting" artifacts, which is impractical for consumer use. Conversely, geometry-based methods that attempt to build 3D meshes often fail on complex structures like foliage or shiny, non-Lambertian surfaces.

Local Light Field Fusion (LLFF) bridges this gap. It is a practical system that turns a few handheld photos into an immersive, navigable 3D scene by expanding each input image into a "Multiplane Image" (MPI) and providing a mathematical "rule of thumb" for image capture, achieving results comparable to traditional light fields while using 4000× fewer input views.

The LLFF Pipeline

The LLFF pipeline consists of three main stages designed to handle irregular view sampling efficiently:

  1. MPI Prediction: The system uses a 3D Convolutional Neural Network (CNN) to "promote" each input 2D image into a Multiplane Image (MPI). An MPI is a stack of semi-transparent RGB-alpha layers at different depths. Unlike previous work using fixed depths, this 3D CNN handles a variable number of depth planes, allowing the system to adapt to different sampling densities.
  2. Local Light Field Fusion: Instead of building one global 3D model, the system treats each MPI as a "local light field." To render a new viewpoint, it identifies the nearest MPIs and renders the scene from their perspectives.
  3. Alpha-Weighted Blending: Renderings from adjacent MPIs are blended using the accumulated alpha (transparency) to weight the blend. This ensures that if one viewpoint sees a pixel occluded in another, the visible version is prioritized, resulting in clean edges and realistic reflections.
The LLFF rendering process: Multiple local MPIs are blended to synthesize a high-quality novel viewpoint.

Prescriptive Sampling Guidelines

A core contribution of this work is providing users with concrete guidance on how to capture a scene.

  • The $D^2$ Advantage: By extending plenoptic sampling theory, the authors show that decomposing a scene into $D$ depth layers allows the spacing between camera shots to increase by a factor of $D$. In a 2D grid, this leads to a $D^2$ reduction in required images.
  • The "64-Pixel Rule": Empirical testing revealed that the pipeline reliably reconstructs scenes if the maximum disparity (pixel shift) of the closest object between adjacent shots is 64 pixels or less. This translates to a simple instruction: capture photos such that the closest object moves no more than approximately 6% of the screen width between shots.
Prescriptive sampling: The system enables high-quality reconstruction from an irregular grid of input views.

Results and Performance

LLFF achieves the perceptual quality of Nyquist-rate sampling while using up to 4000× fewer images. In quantitative comparisons against state-of-the-art methods like Soft3D and Unstructured Lumigraph Rendering (ULR), LLFF consistently scored higher on PSNR and SSIM, and significantly better on LPIPS (a metric tracking human perceptual judgment).

Furthermore, the system is optimized for real-time use. The authors developed a desktop viewer capable of rendering at 60 FPS (1000x700 resolution) and a mobile AR app that guides users through the capture process in real-time, ensuring the 64-pixel rule is met.

Limitations

Despite its robustness, LLFF faces challenges with repetitive textures (like picket fences) or extremely thin structures where MPI prediction may struggle. Additionally, the memory and computational cost of the layered representation scales cubically ($O(W^3)$) with image resolution, presenting a hurdle for 4K or 8K video. Finally, the current implementation assumes a static scene; handling dynamic objects or changing lighting remains an area for future research.

Summary updated

Abstract

We present a practical and robust deep learning solution for capturing and rendering novel views of complex real world scenes for virtual exploration. Previous approaches either require intractably dense view sampling or provide little to no guidance for how users should sample views of a scene to reliably render high-quality novel views. Instead, we propose an algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields. We extend traditional plenoptic sampling theory to derive a bound that specifies precisely how densely users should sample views of a given scene when using our algorithm. In practice, we apply this bound to capture and render views of real world scenes that achieve the perceptual quality of Nyquist rate view sampling while using up to 4000x fewer views. We demonstrate our approach's practicality with an augmented reality smartphone app that guides users to capture input images of a scene and viewers that enable realtime virtual exploration on desktop and mobile platforms.
BibTeX
@article{mildenhall2019local,
  author = {Ben Mildenhall and Pratul P. Srinivasan and Rodrigo Ortiz Cayon and Nima Khademi Kalantari and Ravi Ramamoorthi and Ren Ng and Abhishek Kar},
  title = {{Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines}},
  journal = {ACM Trans. Graphics},
  year = {2019},
  url = {https://openreview.net/forum?id=jB83cCVAKEk},
}
Figure