Rotation-Equivariant Conditional Spherical Neural Fields for Learning a Natural Illumination Prior

James A. D. Gardner, Bernhard Egger, William A. P. Smith

TL;DR

This paper introduces RENI, a novel neural network that learns a "prior" for natural lighting conditions. By building rotation-equivariance directly into the model, RENI can generate realistic High Dynamic Range (HDR) illumination maps and significantly improve inverse rendering tasks, outperforming traditional methods.

What Problem Does This Paper Address?

Accurately representing and understanding lighting in 3D scenes is crucial for realistic computer graphics and computer vision tasks like relighting and object insertion. Real-world lighting is complex and has a very high dynamic range. While traditional methods like Spherical Harmonics (SH) and Spherical Gaussians (SG) offer compact representations, they lack a strong prior for what constitutes "natural" lighting, meaning they can represent any lighting, even physically implausible ones. This makes inverse rendering (inferring lighting from images) an ill-posed problem, as there are too many possible solutions. This paper aims to address this by learning a statistical prior specifically for natural illumination environments, making inverse rendering more robust and enabling more realistic scene synthesis.

Methods

The core of RENI is a Rotation-Equivariant Conditional Spherical Neural Field. Here's a breakdown:

  • Spherical Neural Field: Instead of using traditional image-based representations, RENI models lighting as a continuous function on a sphere. This allows querying the lighting in any direction without the distortions of projections like equirectangular maps.
  • SIREN (Sinusoidal Representation Network): This type of neural network is used as the underlying "field" due to its effectiveness in representing continuous signals with high-frequency details.
  • Vector Neurons Extension: The authors extend the concept of Vector Neurons, a framework for building rotation-equivariance into neural networks. They adapt it to work with spherical signals and specifically focus on SO(2) equivariance. This means the model respects the natural property that while the horizontal rotation of natural light is arbitrary, the "up" direction is canonical (aligned with gravity). This is a key departure from full SO(3) equivariance, which would allow unnatural orientations like the sky at the bottom.
  • Variational Auto-Decoder (VAD): RENI is trained as a VAD. This allows it to learn a generative model of natural illuminations. The encoder learns a distribution (mean and variance) for latent codes representing each environment, and the decoder (the spherical neural field) reconstructs the environment from these codes. The VAD structure also provides a regularized latent space, enabling sampling for new plausible illuminations and acting as a statistical prior for inverse problems.

Datasets

The authors curated a dataset of 1,694 HDR equirectangular images of outdoor, natural illumination environments. They filtered these images to remove those with predominantly unnatural light sources or personally identifiable information. A small subset of 21 images was held back for optimizing latent codes at test time, resulting in a training dataset of 1,673 images.

Key Insights and Ideas

  • Prioritizing Illumination Priors: The paper argues that focusing on learning a prior for illumination itself, rather than just shape or appearance, is a powerful way to tackle the ill-posed nature of inverse rendering.
  • SO(2) Equivariance is Key: Restricting rotation equivariance to SO(2) (rotations around the vertical axis) is crucial for learning a natural illumination prior. Full SO(3) equivariance would allow physically impossible lighting scenarios, diluting the prior's effectiveness.
  • Generative Models for Illumination: Using a VAD framework allows RENI to not only represent existing illuminations but also to generate new, plausible ones, acting as a powerful prior for downstream tasks.
  • High-Frequency Detail Matters: RENI's SIREN-based architecture, combined with its generative capabilities, allows it to capture complex, high-frequency lighting details that traditional SH representations struggle with.

Key Results

  • Improved Generalization: RENI significantly outperforms SH and SG in terms of mean PSNR when fitting to unseen environment maps, especially as latent code dimensionality increases. For a latent dimension of 147, RENI achieves a PSNR of 19.97, compared to 16.44 for SH and 19.26 for SG.
  • Realistic Environment Completion: When given only a partial view of an environment, RENI can hallucinate plausible completions, often accurately estimating sun locations, outperforming SH in realism.
  • Enhanced Inverse Rendering: In an inverse rendering task with a Blinn-Phong shader, RENI produces more natural environment maps and significantly better reconstructions, especially as specularity increases, compared to SH.
  • SO(2) vs. SO(3) vs. Augmentation: The SO(2)-equivariant RENI model performs best, outperforming both SO(3)-equivariant models and models trained solely with data augmentation, while using less data.

Limitations and Future Work

  • Gram Matrix Limitation: The use of a Gram matrix for invariant representation limits the practical size of the latent code. The authors suggest using the invariant layer from Vector Neurons [15] to address this.
  • High-Frequency Detail: While better than SH/SG, the very highest frequency details are still not fully resolved. Larger datasets and network sizes are suggested to improve this.
  • Indoor Scene Performance: RENI's prior is tuned for natural outdoor lighting, leading to decreased performance on indoor scenes. However, it still generalizes reasonably well.
  • Interaction with Other Priors: RENI currently learns an illumination prior independently, not considering its interaction with geometry or texture priors (e.g., the Hollow Face Illusion).
  • SO(3) Equivariance for Dynamic Cameras: For scenarios where the camera's "up" axis doesn't align with gravity, full SO(3) equivariance might be necessary, potentially combined with learning camera pose priors.
  • Future Work: The authors suggest exploring RENI in larger inverse rendering pipelines, using it for LDR to HDR reconstruction, and developing it into a GAN framework for spherical signals.

Summary updated

Abstract

Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance. In this work, we instead focus on a prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, a simplistic prior on the parameters. We propose a conditional neural field representation based on a variational auto-decoder with a SIREN network and, extending Vector Neurons, build equivariance directly into the network. Using this, we develop a rotation-equivariant, high dynamic range (HDR) neural illumination model that is compact and able to express complex, high-frequency features of natural environment maps. Training our model on a curated dataset of 1.6K HDR environment maps of natural scenes, we compare it against traditional representations, demonstrate its applicability for an inverse rendering task and show environment map completion from partial observations. A PyTorch implementation, our dataset and trained models can be found at jadgardner.github.io/RENI.
BibTeX
@inproceedings{gardner2022rotation,
  author = {James A. D. Gardner and Bernhard Egger and William A. P. Smith},
  title = {{Rotation-Equivariant Conditional Spherical Neural Fields for Learning a Natural Illumination Prior}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=cj6K4IWVomU},
}
Figure