Abstract
This paper presents a novel "lazy-learning" approach for editing High Dynamic Range (HDR) environment maps through 2D brush strokes. By representing the environment map as a robust neural implicit function (R-SIREN) and optimizing it via a differentiable renderer, we enable intuitive lighting modifications—such as shifting shadows or introducing reflections—while preserving the visual coherence of the background. Our method avoids the artifacts common in direct pixel manipulation by optimizing in the weight space of the neural representation.
Introduction
Environment maps are essential for realistic lighting in computer graphics but remain challenging to edit. Traditional methods often require manual HDR pixel manipulation, which frequently introduces noise and artifacts. Automating this process via inverse rendering faces three primary hurdles: the sparsity of light sources, the extreme dynamic range of pixel values, and the difficulty of maintaining visual consistency. This work addresses these challenges by parameterizing the environment map as a neural network and optimizing its weights to match user-defined targets.
Methodology
The proposed pipeline shifts the optimization target from individual pixels to the weights of a neural network.
HDR SIREN and R-SIREN
We utilize a Sinusoidal Representation Network (SIREN) to represent the environment map. To accommodate HDR values, the network is trained in a normalized logarithmic space. We employ a dual-loss strategy: one loss ensures structural integrity in log-space, while another focuses on high-intensity peaks in HDR-space. To prevent the optimization from producing "unnatural" maps (e.g., static noise), we introduce Adversarial Weight Perturbation (R-SIREN). This technique forces the model to find smooth regions in the weight space, acting as a powerful implicit prior for natural-looking structures.
Inverse Differentiable Rendering
The system integrates the Mitsuba 3 differentiable renderer. Gradients derived from a target image—where a user has painted desired changes like a new shadow—are backpropagated to update the R-SIREN weights. This process is further stabilized by a regularization suite including luminance stability, $L_1$ sparsity for light sources, and the DISTS perceptual metric.
Results
The R-SIREN approach was evaluated using the PolyHaven dataset, specifically focusing on "PureSky" maps.
Quantitative and Qualitative Performance
Quantitatively, our method outperforms standard pixel-based optimization. For instance, in the "Horse" scene, R-SIREN HDR achieved a PSNR of ~29.7 dB, significantly higher than the ~18.3 dB achieved by pixel optimization. Qualitatively, the method handles complex edits such as moving shadows and adding reflections with high fidelity. Unlike baseline methods that often result in "salt-and-pepper" noise in the sky, our approach maintains visual coherence.
Conclusion
We have demonstrated a lightweight, "on-the-fly" method for environment map editing that leverages the implicit priors of neural networks. While the system currently relies on reflections and may hallucinate light sources in unobserved regions, it provides a robust foundation for intuitive lighting control. Future work will explore extending this framework to 3D model manipulation and incorporating more diverse training data to further refine the local latent space of the neural functions.
Summary updated