Abstract
Estimating scene illumination from limited visual data is a fundamental challenge in inverse rendering. Traditional representations like Spherical Harmonics (SH) and Spherical Gaussians (SG) lack a statistical prior of natural lighting, often failing to capture high-frequency details or realistic sky distributions. We present RENI (Rotation-Equivariant Natural Illumination), a conditional spherical neural field that learns a robust prior for high dynamic range (HDR) outdoor lighting. By incorporating SO(2) equivariance around the vertical axis, RENI ensures that rotated latent codes correspond to physically plausible, rotated environment maps. Our results demonstrate that RENI significantly outperforms SH and SG in both representation accuracy and environment completion tasks.
Introduction
Inverse rendering aims to decompose an image into its constituent parts: shape, material, and lighting. While priors for geometry and reflectance have advanced significantly, lighting models often rely on simplistic mathematical bases. Environment maps—360-degree HDR images—are the gold standard for realistic lighting but are difficult to estimate from partial views. Existing methods like Spherical Harmonics (SH) are often too smooth to represent sharp light sources like the sun, while Spherical Gaussians (SG) lack the efficiency and statistical grounding to distinguish natural skies from random noise. They treat a natural sky and a random jumble of neon noise as equally likely.
Methodology
RENI leverages a Variational Auto-Decoder (VAD) framework to learn a continuous representation of natural light. Instead of discrete pixels, the model uses a SIREN MLP with periodic activations to define a spherical neural field. A key innovation is the enforcement of SO(2) equivariance. Recognizing that natural light has a fixed vertical orientation due to gravity, the authors utilize an invariant transformation that allows the latent code to be rotated around the y-axis while maintaining the structural integrity of the scene. This ensures that if you rotate the latent code, the resulting environment map rotates perfectly without losing its "naturalness."
The model was trained on a curated dataset of 1,694 HDR equirectangular images of outdoor scenes. Training in log-space allows the network to capture the extreme dynamic range—the massive intensity difference between a dark forest and the surface of the sun—ensuring high-frequency details are preserved.
Results and Discussion
RENI demonstrates superior performance in fitting unseen environment maps. Quantitatively, at a tiny latent dimension of $D=27$, RENI achieves a PSNR of 17.02, substantially higher than SH (12.80) and SG (16.04). Qualitatively, RENI reconstructs sharp light sources and realistic horizons where traditional methods produce blurry or artifact-heavy results.
Beyond simple representation, the learned prior enables environment completion. Given a limited field of view, RENI can realistically hallucinate the missing 360-degree environment, accurately predicting the sun's position even when it is out of frame. In inverse rendering tasks, such as estimating light from a rendered specular object, RENI produces much more realistic environment maps than SH, as high-frequency light details are critical for shiny surfaces.
Conclusion
RENI introduces a powerful neural prior for natural illumination by baking geometric symmetries into a neural field architecture. By restricting equivariance to the vertical axis (SO(2)), the model respects the physical reality of our planet, preventing the generation of "upside-down" suns. While currently optimized for outdoor scenes, it sets a new standard for compact, high-fidelity lighting representations. Future work may address the $O(N^2)$ complexity of the invariant transformation and extend the prior to more complex indoor environments where a consistent "up" prior is less prevalent.
Summary updated