Rotation-Equivariant Conditional Spherical Neural Fields for Learning a Natural Illumination Prior

James A D Gardner, Bernhard Egger, William A P Smith

Abstract

Estimating scene illumination from limited visual data is a fundamental challenge in inverse rendering. Traditional representations like Spherical Harmonics (SH) and Spherical Gaussians (SG) lack a statistical prior of natural lighting, often failing to capture high-frequency details or realistic sky distributions. We present RENI (Rotation-Equivariant Natural Illumination), a conditional spherical neural field that learns a robust prior for high dynamic range (HDR) outdoor lighting. By incorporating SO(2) equivariance around the vertical axis, RENI ensures that rotated latent codes correspond to physically plausible, rotated environment maps. Our results demonstrate that RENI significantly outperforms SH and SG in both representation accuracy and environment completion tasks.

Introduction

Inverse rendering aims to decompose an image into its constituent parts: shape, material, and lighting. While priors for geometry and reflectance have advanced significantly, lighting models often rely on simplistic mathematical bases. Environment maps—360-degree HDR images—are the gold standard for realistic lighting but are difficult to estimate from partial views. Existing methods like Spherical Harmonics (SH) are often too smooth to represent sharp light sources like the sun, while Spherical Gaussians (SG) lack the efficiency and statistical grounding to distinguish natural skies from random noise. They treat a natural sky and a random jumble of neon noise as equally likely.

Methodology

RENI leverages a Variational Auto-Decoder (VAD) framework to learn a continuous representation of natural light. Instead of discrete pixels, the model uses a SIREN MLP with periodic activations to define a spherical neural field. A key innovation is the enforcement of SO(2) equivariance. Recognizing that natural light has a fixed vertical orientation due to gravity, the authors utilize an invariant transformation that allows the latent code to be rotated around the y-axis while maintaining the structural integrity of the scene. This ensures that if you rotate the latent code, the resulting environment map rotates perfectly without losing its "naturalness."

Architecture of the Rotation-Equivariant Conditional Spherical Neural Field, illustrating the invariant transformation and the neural field layers.

The model was trained on a curated dataset of 1,694 HDR equirectangular images of outdoor scenes. Training in log-space allows the network to capture the extreme dynamic range—the massive intensity difference between a dark forest and the surface of the sun—ensuring high-frequency details are preserved.

Results and Discussion

RENI demonstrates superior performance in fitting unseen environment maps. Quantitatively, at a tiny latent dimension of $D=27$, RENI achieves a PSNR of 17.02, substantially higher than SH (12.80) and SG (16.04). Qualitatively, RENI reconstructs sharp light sources and realistic horizons where traditional methods produce blurry or artifact-heavy results.

Qualitative comparison of RENI against Spherical Harmonics (SH) and Spherical Gaussians (SG) across different latent dimensions (D=27, 108, 147).

Beyond simple representation, the learned prior enables environment completion. Given a limited field of view, RENI can realistically hallucinate the missing 360-degree environment, accurately predicting the sun's position even when it is out of frame. In inverse rendering tasks, such as estimating light from a rendered specular object, RENI produces much more realistic environment maps than SH, as high-frequency light details are critical for shiny surfaces.

Conclusion

RENI introduces a powerful neural prior for natural illumination by baking geometric symmetries into a neural field architecture. By restricting equivariance to the vertical axis (SO(2)), the model respects the physical reality of our planet, preventing the generation of "upside-down" suns. While currently optimized for outdoor scenes, it sets a new standard for compact, high-fidelity lighting representations. Future work may address the $O(N^2)$ complexity of the invariant transformation and extend the prior to more complex indoor environments where a consistent "up" prior is less prevalent.

Summary updated

Abstract

Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance. In this work, we instead focus on a prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, a simplistic prior on the parameters. We propose a conditional neural field representation based on a variational auto-decoder with a SIREN network and, extending Vector Neurons, build equivariance directly into the network. Using this, we develop a rotation-equivariant, high dynamic range (HDR) neural illumination model that is compact and able to express complex, high-frequency features of natural environment maps. Training our model on a curated dataset of 1.6K HDR environment maps of natural scenes, we compare it against traditional representations, demonstrate its applicability for an inverse rendering task and show environment map completion from partial observations.
BibTeX
@inproceedings{gardner2022rotation,
  author = {James A D Gardner and Bernhard Egger and William A P Smith},
  title = {{Rotation-Equivariant Conditional Spherical Neural Fields for Learning a Natural Illumination Prior}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=cj6K4IWVomU},
}
Figure