Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition

Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, Hendrik P. A. Lensch

TL;DR

Neural-PIL is a novel framework that decomposes a scene into its 3D shape, material properties (BRDF), and illumination from a set of images. By replacing expensive rendering integrals with a fast neural network query and using smooth manifold priors, it achieves higher-fidelity relighting and faster optimization than previous state-of-the-art methods like NeRD.

What Problem Does This Paper Address?

In computer vision, "inverse rendering" is the holy grail: taking a set of 2D images and figuring out the underlying 3D shape, what the objects are made of (reflectance), and how the scene is lit. While Neural Radiance Fields (NeRF) have mastered "view synthesis," they don't actually understand the scene; they just learn to predict the color of a pixel from a specific angle. They can't tell the difference between a white object in red light and a red object in white light.

Existing attempts to solve this decomposition (like NeRD or PhySG) often use simplified mathematical models called Spherical Gaussians (SGs) to represent light. While SGs are computationally efficient, they are "blurry"—they can't capture the sharp details of a window or a studio light panel. Furthermore, calculating how light reflects off a surface usually requires solving a complex integral (summing up all light hitting a point), which is incredibly slow for deep learning optimization.

Methods

The authors introduce two primary innovations to solve these bottlenecks:

  1. Neural Pre-Integrated Lighting (Neural-PIL): Instead of performing a "brute-force" numerical integration of light during rendering, the authors train a specialized MLP to learn the result of that integral. By querying this network with a latent lighting code, surface roughness, and a reflection direction, the system gets the "pre-integrated" light value instantly. This is inspired by real-time game engine techniques but moved into a differentiable neural framework.
Pre-integrated lighting maps at varying roughness levels.
  1. Smooth Manifold Auto-Encoders (SMAE): Inverse rendering is "ill-posed"—there are many combinations of light and material that could result in the same pixel color. To constrain the search, the authors learn low-dimensional "priors" for both materials (BRDFs) and lighting. Crucially, these auto-encoders are designed to be "smooth," meaning that small changes in the latent code lead to small, plausible changes in the material or light, which makes gradient-based optimization much more stable.

The overall architecture uses a coarse-to-fine approach: a coarse NeRF-like network finds the general geometry, and a fine decomposition network extracts the detailed shape and material properties.

Datasets

The researchers evaluated their model on several benchmarks:

  • Synthetic Datasets: Chair, Globe, and Car models (originally from the NeRD paper) used to measure ground-truth accuracy in decomposition.
  • Real-World Datasets: Multi-view captures including "Cape" (fixed light), "Head" (varying light), and "Gnome" (varying light).
  • Pre-training Data: To train the SMAEs and Neural-PIL, they used 320 high-quality environment maps from HDRi Haven and a large collection of real-world material samples.

Key Insights and Ideas

  • Query vs. Integral: The biggest "aha moment" is realizing that the bottleneck of inverse rendering isn't just the representation, but the math of the rendering equation itself. By turning an integral into a network query, they achieve a massive speedup in the rendering component (~100x faster than SG-based rendering).
  • Expressiveness Matters: Unlike Spherical Gaussians, which act like low-pass filters, Neural-PIL can represent high-frequency lighting details. This allows the model to recover sharp reflections on shiny objects that previous methods would simply blur out.
  • Smoothness as a Regularizer: Standard latent spaces can be "jagged," causing optimization to get stuck in local minima. By enforcing a smooth manifold, the authors ensure the optimizer can "glide" toward the correct physical properties of the scene.

Key Results

  • Higher Fidelity: Neural-PIL consistently outperformed NeRD and NeRF in both synthetic and real-world scenarios. In BRDF decomposition tasks, it showed significantly higher PSNR (Peak Signal-to-Noise Ratio) for diffuse and roughness parameters.
  • Speed: Rendering a million samples with Neural-PIL takes just 1.86 ms, compared to 210 ms for a 24-lobe Spherical Gaussian representation.
  • Relighting: Because the decomposition is more accurate, the model can perform "illumination transfer"—taking an object from one scene and realistically placing it in another—with much more convincing results than prior art.
Full pipeline from training images to decomposition and relighting.

Limitations and Future Work

Despite the progress, some challenges remain:

  • Single Bounce Only: The current model assumes light hits a surface and bounces once to the camera. It does not account for "inter-reflections" (light bouncing between two objects) or complex shadows.
  • Ambiguity: While the SMAE priors help, the problem remains fundamentally difficult. The model can still occasionally converge to a "wrong" solution that happens to look correct from the training viewpoints.
  • Mirror Reflections: While better than SGs, the model still struggles with perfect, mirror-like reflections, which are limited by the resolution of the neural representation.

Future work may involve incorporating more complex light transport (like global illumination) and improving the resolution of the recovered environment maps.

Summary updated

Abstract

Decomposing a scene into its shape, reflectance and illumination is a fundamental problem in computer vision and graphics. Neural approaches such as NeRF have achieved remarkable success in view synthesis, but do not explicitly perform decomposition and instead operate exclusively on radiance (the product of reflectance and illumination). Extensions to NeRF, such as NeRD, can perform decomposition but struggle to accurately recover detailed illumination, thereby significantly limiting realism. We propose a novel reflectance decomposition network that can estimate shape, BRDF, and per-image illumination given a set of object images captured under varying illumination. Our key technique is a novel illumination integration network called Neural-PIL that replaces a costly illumination integral operation in the rendering with a simple network query. In addition, we also learn deep low-dimensional priors on BRDF and illumination representations using novel smooth manifold auto-encoders. Our decompositions can result in considerably better BRDF and light estimates enabling more accurate novel view-synthesis and relighting compared to prior art. Project page: https://markboss.me/publication/2021-neural-pil/
BibTeX
@inproceedings{boss2021neural,
  author = {Mark Boss and Varun Jampani and Raphael Braun and Ce Liu and Jonathan T. Barron and Hendrik P. A. Lensch},
  title = {{Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=KkZIaLiwf2U3},
}
Figure